profile
viewpoint

keras-team/keras 46978

Deep Learning for humans

fchollet/deep-learning-with-python-notebooks 9737

Jupyter notebooks for the code samples of the book "Deep Learning with Python"

fchollet/deep-learning-models 6039

Keras code and weights files for popular deep learning models.

fchollet/keras-resources 3008

Directory of tutorials and open-source code repositories for working with Keras, the Python deep learning library

keras-team/keras-tuner 1744

Hyperparameter tuning for humans

keras-team/keras-contrib 1332

Keras community contributions

fchollet/ARC 1056

The Abstraction and Reasoning Corpus

keras-team/keras-docs-zh 570

Chinese (zh-cn) translation of the Keras documentation.

fchollet/hualos 356

Keras Total Visualization project

tensorflow/estimator 157

TensorFlow Estimator

issue commenttensorflow/tensorflow

Add a Model.fit-like API for seq2seq models

Consider creating a subclass of Model that overrides the training step (see method _train_step) or the fit loop. This is now very easy to do. You'll be able to use such a Model subclass to create new models either via the Functional API or via subclassing.

luozhouyang

comment created time in 3 days

push eventfchollet/ARC

Samuel

commit sha df1bef60865be84802892e44c84f536bf6ff3baa

display task name (and idx) upon random selection

view details

Samuel

commit sha d259771ffe0d6ce5181c8fabb3752f1828ad6096

remove parentheses to declutter display

view details

Samuel

commit sha d9b2f867317f165db0a6cc9d8b8af67cd98f7c1d

organize labels

view details

Samuel

commit sha 27d91f73fda784bf59a0d90b12ccfc1842bbee2f

task-from-file and random-task both display task name

view details

Samuel

commit sha 8c5933aa59bada3430c2d0f2fe12e991d5477cba

un-abbreviated variable names

view details

François Chollet

commit sha 6cd7d2640304256e85dca20ee44a2b452c8c9e43

Merge pull request #35 from bohrium/verbose_js_interface Verbose js interface

view details

push time in 4 days

PR merged fchollet/ARC

Verbose js interface

To aid human exploration of the data set, we show the task's name (and its index in the training set) whenever we randomly select a task.

New: arc-interface-task-name

Old: arc-interface-old

+36 -18

0 comment

2 changed files

bohrium

pr closed time in 4 days

Pull request review commentfchollet/ARC

Verbose js interface

 function loadJSONTask(train, test) {     $('#total_test_input_count_display').html(test.length); } +function display_task_nm(task_nm, idx, nb_tasks) {

Please use full variable names rather than abbreviations. e.g. "task_name" or "name"

bohrium

comment created time in 5 days

Pull request review commentfchollet/ARC

Add cropping functionality

 $(document).ready(function () {                 errorMsg('Can only paste at a specific location; only select *one* cell as paste destination.');             }         }+        // Crop code+        if (event.which == 82) {+            // Press R

Ok, but this means the feature is not discoverable by a user other than you... the R key should be advertised (in the same place as the C and V commands)

eeegnu

comment created time in 7 days

Pull request review commentfchollet/ARC

Add cropping functionality

 $(document).ready(function () {                 errorMsg('Can only paste at a specific location; only select *one* cell as paste destination.');             }         }+        // Crop code+        if (event.which == 82) {+            // Press R+            selected = $('.ui-selected');+            if (selected.length == 0) {+                return;+            }++            COPY_PASTE_DATA = [];+            for (var i = 0; i < selected.length; i++) {+                x = parseInt($(selected[i]).attr('x'));+                y = parseInt($(selected[i]).attr('y'));+                symbol = parseInt($(selected[i]).attr('symbol'));+                COPY_PASTE_DATA.push([x, y, symbol]);+            }++            xs = new Array();+            ys = new Array();+            symbols = new Array();++            for (var i = 0; i < COPY_PASTE_DATA.length; i++) {+                xs.push(COPY_PASTE_DATA[i][0]);+                ys.push(COPY_PASTE_DATA[i][1]);+                symbols.push(COPY_PASTE_DATA[i][2]);+            }++            minx = Math.min(...xs);+            miny = Math.min(...ys);+            for (var i = 0; i < xs.length; i++) {+                x = xs[i];+                y = ys[i];+                symbol = symbols[i];+                newx = x - minx;+                newy = y - miny;+                res = jqGrid.find('[x="' + newx + '"][y="' + newy + '"] ');+                if (res.length == 1) {+                    cell = $(res[0]);+                    setCellSymbol(cell, symbol);+                }+            }+            $('#output_grid_size').val((xs[xs.length - 1] - minx + 1) + 'x' + (ys[ys.length - 1] - miny + 1));

Rather than manipulating the HTML directly, you should fill in CURRENT_OUTPUT_GRID then sync via syncFromDataGridToEditionGrid().

eeegnu

comment created time in 7 days

push eventfchollet/ARC

Gunnar Aastrand Grimnes

commit sha a902863393ac9ea4324f14801ea2b6eed8749e8f

fix example 3 for training/868de0fa the 4th square, to be filled in brown was moved one line down in the output. There should not be an empty line between the 3rd and 4th.

view details

François Chollet

commit sha b980340ee8159f7e357f15ca440c343d84a39c44

Merge pull request #45 from gromgull/868de0fa Fix example 3 for training/868de0fa

view details

push time in 7 days

PR merged fchollet/ARC

Fix example 3 for training/868de0fa

the 4th square, to be filled in brown was moved one line down in the output. There should not be an empty line between the 3rd and 4th.

Before: image

After: image

+1 -1

0 comment

1 changed file

gromgull

pr closed time in 7 days

push eventfchollet/ARC

skewwhiff

commit sha 33782748fe7ef0fa4d8ff3a3cc7d57eff1ae992c

Fixed training label mismatch This training example has its input and output interchanged in the last example. Fixed them.

view details

François Chollet

commit sha 915ea1fcc5dc5923ef662da54b17fd74c649fd79

Merge pull request #37 from skewwhiff/patch-1 Fixed training label mismatch

view details

push time in 10 days

PR merged fchollet/ARC

Fixed training label mismatch

This training example has its input and output interchanged in the last example. Fixed them.

The last train example pair should be:

{"input": [
[0, 0, 0, 0, 3, 0, 0, 0, 0, 5], 
[0, 0, 0, 0, 3, 0, 0, 0, 0, 0], 
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3], 
:
:
], "output": [
[0, 0, 0, 3, 0, 0, 0, 0, 0, 0], 
[0, 0, 0, 3, 0, 0, 0, 0, 0, 0], 
[0, 0, 0, 3, 0, 0, 0, 0, 0, 0],
 [3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
: 
: ]
]}]

instead of

{"input": [
[0, 0, 0, 0, 3, 0, 0, 0, 0, 0], 
[0, 0, 0, 0, 3, 0, 0, 0, 0, 0], 
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
: 
: ], 
"output": [
[0, 0, 0, 3, 0, 0, 0, 0, 0, 5], 
[0, 0, 0, 3, 0, 0, 0, 0, 0, 0], 
[0, 0, 0, 3, 0, 0, 0, 0, 0, 0], 
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3], 
: 
: 
: ]}]
+1 -1

1 comment

1 changed file

skewwhiff

pr closed time in 10 days

pull request commentfchollet/ARC

Fixed training label mismatch

Thank you for the fix.

skewwhiff

comment created time in 10 days

push eventfchollet/ARC

Benjamin Beyret

commit sha c257f70ff6c840a6fd59b83636061677fefcef3a

fix example 1 training/42a50994.json

view details

François Chollet

commit sha de4d44a01f0d8eb09dd7724b01e0574295d4f42e

Merge pull request #43 from beyretb/master Fix example 1 in training/42a50994.json

view details

push time in 10 days

PR merged fchollet/ARC

Fix example 1 in training/42a50994.json

Example 1 output is missing 2 blue squares in the top left corner (according to the other 3 examples these two should be there as they're "connected")

Before fix: image

After fix: image

+1 -1

0 comment

1 changed file

beyretb

pr closed time in 10 days

push eventfchollet/ARC

dmauskop

commit sha 9296365e4351dc7d927938271ace79ee49e80990

Prevent infinite loop in floodfill corner case Prevent infinite loop that results if user tries to redundantly floodfill a section of the grid with the section's existing color.

view details

François Chollet

commit sha a2081981bba2641f19b9db7aefb1cf0c080aeb1f

Merge pull request #42 from dmauskop/patch-2 Prevent infinite loop in floodfill corner case

view details

push time in 10 days

PR merged fchollet/ARC

Prevent infinite loop in floodfill corner case

Prevent infinite loop that results if user tries to redundantly floodfill a section of the grid with the section's existing color.

+5 -1

0 comment

1 changed file

dmauskop

pr closed time in 10 days

push eventfchollet/ARC

dmauskop

commit sha 3a1f0e70da731ef1a672f1c689369ac457250a94

Make sure symbol_0 is selected by default Change the initial class for symbol_0 (black) from "selected" to "selected-symbol-preview" so it's selected by default. Without this change there's a subtle bug where the user can appear to be making edits that aren't actually reflected in the underlying grid. Consider input 7f4411d.json. If the user copies the input grid and then starts clicking on the extra cells without selecting a color, the cells will change to black. When the user submits a grid that appears to be correct, they will be told their answer is incorrect because the changes weren't reflected in the underlying grid.

view details

François Chollet

commit sha c2f6317ac24afd8a8f552de132ceccfba4ece3df

Merge pull request #41 from dmauskop/patch-1 Make sure symbol_0 is selected by default

view details

push time in 10 days

PR merged fchollet/ARC

Make sure symbol_0 is selected by default

Change the initial class for symbol_0 (black) from "selected" to "selected-symbol-preview" so it's selected by default. Without this change there's a subtle bug where the user can appear to be making edits that aren't actually reflected in the underlying grid.

Consider input 7f4411d.json. If the user copies the input grid and then starts clicking on the extra cells without selecting a color, the cells will change to black. When the user submits a grid that appears to be correct, they will be told their answer is incorrect because the changes weren't reflected in the underlying grid.

+1 -1

0 comment

1 changed file

dmauskop

pr closed time in 10 days

pull request commentfchollet/ARC

Update d687bc17.json

Thank you for the fix.

ngdelamo

comment created time in 10 days

push eventfchollet/ARC

Nacho G. del Amo

commit sha 4ca45088ec379d14bf73e731a35a8aee2a9f2d06

Update d687bc17.json Correction for the 3rd example of the train set on task d687bc17

view details

François Chollet

commit sha ce7935624685d24f775666f3ab885b3deaf66371

Merge pull request #39 from ngdelamo/master Update d687bc17.json

view details

push time in 10 days

PR merged fchollet/ARC

Update d687bc17.json

Correction for the 3rd example of the train set on task d687bc17. Please, see the attached image for the explanation of the changed values.

image

+1 -1

0 comment

1 changed file

ngdelamo

pr closed time in 10 days

push eventfchollet/ARC

Koke_Cacao

commit sha db2742b9f69f3c5e1abb7f4a0ada33e59c74b025

Fixed 2nd output for 1b60fb0c.json

view details

Hanke Chen

commit sha 0c5d3c64a4bf941f0c82a78206aa5f02a8ec7e2c

fix 11852cab.json

view details

François Chollet

commit sha 724149b62816097a76a036a0120753a97ec20a58

Merge pull request #33 from KokeCacao/master Fix example 2 in training/1b60fb0c.json and example 1 in training/11852cab.json

view details

push time in 11 days

PR merged fchollet/ARC

Fix example 2 in training/1b60fb0c.json and example 1 in training/11852cab.json

1b60fb0c.json

Before:

image

After:

image

11852cab.json

Before:

image

After:

image

Thank you for giving us interesting problems to solve!

+2 -2

0 comment

2 changed files

KokeCacao

pr closed time in 11 days

pull request commentfchollet/ARC

Fix example 1 of training data 82819916

Thanks for the fix!

GuyAglionby

comment created time in 12 days

push eventfchollet/ARC

Guy Aglionby

commit sha 8c4d2b58d97f551b8f872fea878926474d210a3a

Fix example 1 of training data 82819916

view details

François Chollet

commit sha 05571aec87a81f7eb8a1071a5a5363981ec10807

Merge pull request #32 from GuyAglionby/master Fix example 1 of training data 82819916

view details

push time in 12 days

PR merged fchollet/ARC

Fix example 1 of training data 82819916

Screen Shot 2020-02-14 at 20 42 56

The first example appears to break the pattern (i.e. red pixel the arrow points to). This change makes that pixel yellow.

+1 -1

0 comment

1 changed file

GuyAglionby

pr closed time in 12 days

push eventfchollet/ARC

Benjamin Beyret

commit sha 328384512f8b5a1bb5655d4c0f02e2947efe383e

fix training/10fcaaa3.json example 2

view details

Benjamin Beyret

commit sha 47f7b035fa488b42e26b8a85762e9f3b0ed379f0

fix training/10fcaaa3.json example 2

view details

François Chollet

commit sha edb16b44ab6c4e73e71b4bdf4b69a53e843a9adf

Merge pull request #31 from beyretb/master Fix example 2 in training/10fcaaa3.json

view details

push time in 12 days

PR merged fchollet/ARC

Fix example 2 in training/10fcaaa3.json

Issue raised on kaggle: https://www.kaggle.com/c/abstraction-and-reasoning-challenge/discussion/130405

Example 2 in training/10fcaaa3.json output is missing a blue square fifth row, rightmost square

Before merge:

image

After merge:

image

+1 -1

1 comment

1 changed file

beyretb

pr closed time in 12 days

pull request commentfchollet/ARC

Fix example 2 in training/10fcaaa3.json

Yes, you are right. Thanks for the fix.

beyretb

comment created time in 12 days

pull request commentkeras-team/governance

RFC: Keras Grouped Convolution

I am happy to send a PR to TensorFlow

Please do.

lgeiger

comment created time in 13 days

PR closed keras-team/governance

RFC: Keras Grouped Convolution

This small proposal aims to add support for grouped convolution to the keras.layers.Conv{1,2,3}D API.

Grouped convolutions are a special case of sparsely connected convolutions and have been successfully used in ResNeXt, ShuffleNet, CondenseNet and many follow up works.

This feature is supported by other frameworks like Caffe or PyTorch and has been requested many times in the TensorFlow and Keras community (https://github.com/tensorflow/tensorflow/issues/3332, https://github.com/tensorflow/tensorflow/issues/11662, https://github.com/keras-team/keras/issues/3334, https://github.com/tensorflow/tensorflow/pull/10482).

Grouped convolutions are now supported in TensorFlow core via CUDNN 7 on GPUs (https://github.com/tensorflow/tensorflow/pull/25818) and inside XLA compiled functions on CPU and GPUs, but there is no support via the Keras layers API yet. Adding support for grouped convolutions should make it easier for users to implement above mentioned models with TensorFlow and Keras in a fast way.

🎨rendered version 🎨

+62 -0

10 comments

1 changed file

lgeiger

pr closed time in 13 days

pull request commentkeras-team/governance

RFC: Keras Grouped Convolution

Thank you for the proposal and the work. We'll carry on the discussion in the TF PR. Since it's a small API change, and it isn't controversial, we don't need a public design review meeting.

lgeiger

comment created time in 13 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import os+import sys++from . import containerize+from . import deploy+from . import gcp+from . import machine_config+from . import package+from . import preprocess+from . import validate+++# Flag which indicates whether current process is running in a cloud+# environment created by the `cloud.run` API.+_IS_RUNNING_REMOTELY = False+++def _is_running_remotely():+    return _IS_RUNNING_REMOTELY+++def _set_running_remotely(value):+    global _IS_RUNNING_REMOTELY+    _IS_RUNNING_REMOTELY = value+++def run(+    entry_point,+    requirements_txt=None,+    distribution_strategy='auto',+    docker_base_image=None,+    chief_config='auto',+    worker_config='auto',+    worker_count=0,+    region=None,+    entry_point_args=None,+    stream_logs=False,+):

You can move ): to the line above

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):+    if accl_type == 'CPU':+        return 'ACCELERATOR_TYPE_UNSPECIFIED'+    if accl_type == 'K80':+        return 'NVIDIA_TESLA_K80'+    if accl_type == 'P100':+        return 'NVIDIA_TESLA_P100'+    if accl_type == 'V100':+        return 'NVIDIA_TESLA_V100'+    if accl_type == 'P4':+        return 'NVIDIA_TESLA_P4'+    if accl_type == 'T4':+        return 'NVIDIA_TESLA_T4'+    else:+        raise ValueError('Invalid accelerator type.')+++def get_machine_type(cpu_cores, memory):

This is also more simply structured as a dict and a get

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import os+import tempfile++from .machine_config import AcceleratorType+++def get_startup_script(entry_point,

Please add docstring:

  • what does the startup script do?
  • what are the arguments?
pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import tarfile+import tempfile+++def get_tarball(file_location_map):

Please add docstring (structure of file_location_map is not easy to determine)

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import random+import string+import subprocess++from googleapiclient import discovery+from googleapiclient import errors++from . import gcp+++def deploy_job(region, image_uri, chief_config, worker_count, worker_config,+               entry_point_args, enable_stream_logs):+    job_name = _get_name()+    project_id = gcp.get_project_name()+    ml_apis = discovery.build('ml', 'v1')++    request_dict = _create_request_dict(+        job_name, region, image_uri, chief_config, worker_count, worker_config,+        entry_point_args)+    try:+        response = ml_apis.projects().jobs().create(+            parent='projects/{}'.format(project_id),+            body=request_dict+        ).execute()+        print('Job submitted successfully.')+        _print_logs_info(job_name, project_id)+        # TODO(psv): Add support for streaming logs.+    except errors.HttpError as err:+        print('There was an error submitting the job.')+        print(err._get_reason())+    return job_name+++def _create_request_dict(job_name, region, image_uri, chief_config,+                         worker_count, worker_config, entry_point_args):+    trainingInput = {}

An object instance should have a snake case name (applicable throughout this file)

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import random+import string+import subprocess++from googleapiclient import discovery+from googleapiclient import errors++from . import gcp+++def deploy_job(region, image_uri, chief_config, worker_count, worker_config,+               entry_point_args, enable_stream_logs):+    job_name = _get_name()+    project_id = gcp.get_project_name()+    ml_apis = discovery.build('ml', 'v1')++    request_dict = _create_request_dict(+        job_name, region, image_uri, chief_config, worker_count, worker_config,+        entry_point_args)+    try:+        response = ml_apis.projects().jobs().create(+            parent='projects/{}'.format(project_id),+            body=request_dict+        ).execute()+        print('Job submitted successfully.')+        _print_logs_info(job_name, project_id)+        # TODO(psv): Add support for streaming logs.+    except errors.HttpError as err:+        print('There was an error submitting the job.')

Prefer re-raising rather than printing (so that the caller of the code can except)

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import os+import sys++from . import containerize+from . import deploy+from . import gcp+from . import machine_config+from . import package+from . import preprocess+from . import validate+++# Flag which indicates whether current process is running in a cloud+# environment created by the `cloud.run` API.+_IS_RUNNING_REMOTELY = False+++def _is_running_remotely():+    return _IS_RUNNING_REMOTELY+++def _set_running_remotely(value):+    global _IS_RUNNING_REMOTELY+    _IS_RUNNING_REMOTELY = value+++def run(+    entry_point,+    requirements_txt=None,+    distribution_strategy='auto',+    docker_base_image=None,+    chief_config='auto',+    worker_config='auto',+    worker_count=0,+    region=None,+    entry_point_args=None,+    stream_logs=False,+):+    """Runs your Tensorflow code in Google Cloud Platform.++    # Arguments:

You use sometimes Args:, and here, # Arguments:. I guess we should prefer Args:; in any case we should be consistent.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):+    if accl_type == 'CPU':+        return 'ACCELERATOR_TYPE_UNSPECIFIED'+    if accl_type == 'K80':+        return 'NVIDIA_TESLA_K80'+    if accl_type == 'P100':+        return 'NVIDIA_TESLA_P100'+    if accl_type == 'V100':+        return 'NVIDIA_TESLA_V100'+    if accl_type == 'P4':+        return 'NVIDIA_TESLA_P4'+    if accl_type == 'T4':+        return 'NVIDIA_TESLA_T4'+    else:+        raise ValueError('Invalid accelerator type.')+++def get_machine_type(cpu_cores, memory):+    config = (cpu_cores, memory)+    if config == (4, 15):+        return 'n1-standard-4'+    if config == (8, 30):+        return 'n1-standard-8'+    if config == (16, 60):+        return 'n1-standard-16'+    if config == (32, 120):+        return 'n1-standard-32'+    if config == (64, 240):+        return 'n1-standard-64'+    if config == (96, 360):+        return 'n1-standard-96'+    if config == (2, 13):+        return 'n1-highmem-2'+    if config == (4, 26):+        return 'n1-highmem-4'+    if config == (8, 52):+        return 'n1-highmem-8'+    if config == (16, 104):+        return 'n1-highmem-16'+    if config == (32, 208):+        return 'n1-highmem-32'+    if config == (64, 416):+        return 'n1-highmem-64'+    if config == (96, 624):+        return 'n1-highmem-96'+    if config == (16, 14.4):+        return 'n1-highcpu-16'+    if config == (32, 28.8):+        return 'n1-highcpu-32'+    if config == (64, 57.6):+        return 'n1-highcpu-64'+    if config == (96, 86.4):+        return 'n1-highcpu-96'+    else:+        raise ValueError('Invalid machine type.')

This should 1) print what the user passed, 2) offer the list of valid options (e.g. via a link to docs, since the actual list is long)

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):+    if accl_type == 'CPU':+        return 'ACCELERATOR_TYPE_UNSPECIFIED'+    if accl_type == 'K80':+        return 'NVIDIA_TESLA_K80'+    if accl_type == 'P100':+        return 'NVIDIA_TESLA_P100'+    if accl_type == 'V100':+        return 'NVIDIA_TESLA_V100'+    if accl_type == 'P4':+        return 'NVIDIA_TESLA_P4'+    if accl_type == 'T4':+        return 'NVIDIA_TESLA_T4'+    else:+        raise ValueError('Invalid accelerator type.')

This should 1) print what the user passed, 2) offer the list of valid options

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import os+import sys++from . import containerize+from . import deploy+from . import gcp+from . import machine_config+from . import package+from . import preprocess+from . import validate+++# Flag which indicates whether current process is running in a cloud+# environment created by the `cloud.run` API.+_IS_RUNNING_REMOTELY = False+++def _is_running_remotely():+    return _IS_RUNNING_REMOTELY+++def _set_running_remotely(value):+    global _IS_RUNNING_REMOTELY+    _IS_RUNNING_REMOTELY = value+++def run(+    entry_point,+    requirements_txt=None,+    distribution_strategy='auto',+    docker_base_image=None,+    chief_config='auto',+    worker_config='auto',+    worker_count=0,+    region=None,+    entry_point_args=None,+    stream_logs=False,+):+    """Runs your Tensorflow code in Google Cloud Platform.++    # Arguments:+        entry_point: String. Python file path to the file that contains the+            TensorFlow code.+            Note: This path must be in the current working directory tree.+            Example: 'train.py', 'training/mnist.py'+        requirements_txt: Optional string. File path to requirements.txt file+            containing aditionally pip dependencies if any.+            Note: This path must be in the current working directory tree.+            Example: 'requirements.txt', 'deps/reqs.txt'+        distribution_strategy: 'auto' or None. Defaults to 'auto'.+            'auto' means we will take care of creating a Tensorflow+            distribution strategy instance based on the machine configurations+            you have provided using the `chief_config`, `worker_config` and+            `worker_count` params.+            - If the number of workers > 0, we will use+                `tf.distribute.experimental.MultiWorkerMirroredStrategy`.+            - If number of GPUs > 0, we will use+                `tf.distribute.MirroredStrategy`+            If you have created a distribution strategy instance in your script+            already, please set `distribution_stratgey` as None here.+            For example, if you are using `tf.keras` custom training loops,+            you will need to create a strategy in the script for distributing+            the dataset.+        docker_base_image: Optional base docker image to use. Defaults to None.+            Example: 'gcr.io/my_gcp_project/deep_learning:v2'+            If a base docker image is not provided here, we will use a+            Tensorflow docker image (https://www.tensorflow.org/install/docker)+            as the base image. The version of TensorFlow and Python in that+            case will match your local environment.+        chief_config: Optional `MachineConfig` that represents the+            configuration for the chief worker in a distribution cluster.+            Defaults to 'auto'. 'auto' maps to a standard gpu config such as+            `COMMON_MACHINE_CONFIGS.P100_1X` (8 cpu cores, 30GB memory,+            1 Nvidia Tesla P100).+        worker_config: Optional `MachineConfig` that represents the+            configuration for the general workers in a distribution cluster.+            Defaults to 'auto'. 'auto' maps to a standard gpu config such as+            `COMMON_MACHINE_CONFIGS.P100_1X` (8 cpu cores, 30GB memory,+            1 Nvidia Tesla P100).+        worker_count: Optional integer that represents the number of general+            workers in a distribution cluster. Defaults to 0. This count does+            not include the chief worker.+        region: Optional string. Cloud region in which to submit the+            job. Defaults to 'us-central1' for GCP.+        entry_point_args: Optional list of strings. Defaults to None.+            Command line arguments to pass to the `entry_point` program.+        stream_logs: Boolean flag which when enabled streams logs back from+            the cloud job.+    """+    # If code is triggered in a cloud environment, do nothing.+    if _is_running_remotely():+        return+    _set_running_remotely(True)++    # Get defaults.+    if chief_config == 'auto':+        chief_config = machine_config.COMMON_MACHINE_CONFIGS['P100_1X']+    if worker_config == 'auto':+        worker_config = machine_config.COMMON_MACHINE_CONFIGS['P100_1X']+    region = region or gcp.get_region()+    dst_path_prefix = '/app/'+    docker_registry = 'gcr.io/{}'.format(gcp.get_project_name())++    # Run validations.+    validate.validate(+        entry_point, distribution_strategy, requirements_txt,+        chief_config, worker_config, worker_count, region,+        entry_point_args, stream_logs)++    # Create the script to run (starter_script).+    # Make the `entry_point` cloud and distribution ready.+    startup_script = preprocess.get_startup_script(+        entry_point, chief_config, worker_count, distribution_strategy)++    # Get all the files, that we need to package, mapped to the dst location.+    # This will include the startup script, requirements_txt, dockerfile,+    # files in the entry_point dir.+    dockerfile, file_map = containerize.get_file_map(+        entry_point, startup_script, chief_config, requirements_txt,+        dst_path_prefix, docker_base_image)++    # Create a tarball with the files.+    tarball = package.get_tarball(file_map)++    # Create docker image.+    docker_img = containerize.get_docker_image(docker_registry, tarball)++    # Delete all the temporary files we created.

Note that we may not necessarily have write access to create temp files. This is not necessarily a meaningful issue (since it's a rare case) , but it's something to keep in mind.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):

This is more simply structured as a dict and a get

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import random+import string+import subprocess++from googleapiclient import discovery+from googleapiclient import errors++from . import gcp+++def deploy_job(region, image_uri, chief_config, worker_count, worker_config,+               entry_point_args, enable_stream_logs):+    job_name = _get_name()+    project_id = gcp.get_project_name()+    ml_apis = discovery.build('ml', 'v1')++    request_dict = _create_request_dict(+        job_name, region, image_uri, chief_config, worker_count, worker_config,+        entry_point_args)+    try:+        response = ml_apis.projects().jobs().create(+            parent='projects/{}'.format(project_id),+            body=request_dict+        ).execute()+        print('Job submitted successfully.')+        _print_logs_info(job_name, project_id)+        # TODO(psv): Add support for streaming logs.+    except errors.HttpError as err:+        print('There was an error submitting the job.')+        print(err._get_reason())+    return job_name+++def _create_request_dict(job_name, region, image_uri, chief_config,+                         worker_count, worker_config, entry_point_args):+    trainingInput = {}+    trainingInput['region'] = region+    trainingInput['scaleTier'] = 'custom'+    trainingInput['masterType'] = gcp.get_machine_type(+        chief_config.cpu_cores, chief_config.memory)++    # Set master config+    masterConfig = {}+    masterConfig['imageUri'] = image_uri+    masterConfig['acceleratorConfig'] = {}+    masterConfig['acceleratorConfig']['count'] = str(+        chief_config.accelerator_count)+    masterConfig['acceleratorConfig']['type'] = gcp.get_accelerator_type(+        chief_config.accelerator_type.value)++    trainingInput['masterConfig'] = masterConfig+    trainingInput['workerCount'] = str(worker_count)++    if worker_count > 0:+        trainingInput['workerType'] = gcp.get_machine_type(+            worker_config.cpu_cores, worker_config.memory)++        workerConfig = {}+        workerConfig['imageUri'] = image_uri+        workerConfig['acceleratorConfig'] = {}+        workerConfig['acceleratorConfig']['count'] = str(+            worker_config.accelerator_count)+        workerConfig['acceleratorConfig']['type'] = gcp.get_accelerator_type(+            worker_config.accelerator_type.value)+        trainingInput['workerConfig'] = workerConfig++    if entry_point_args is not None:+        trainingInput['args'] = entry_point_args+    trainingInput['use_chief_in_tf_config'] = True+    request_dict = {}+    request_dict['jobId'] = job_name+    request_dict['trainingInput'] = trainingInput+    return request_dict+++def _print_logs_info(job_name, project_id):+    print('Your job ID is: ', job_name)+    print('Please access your job logs at the following URL:')+    print('https://console.cloud.google.com/mlengine/jobs/{}?project={}'+          .format(job_name, project_id))+++def _get_name():+    unique_tag = ''.join(random.choice(

Likewise, we should something more random here.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')

Raise a RuntimeError here. In general, never raise an unspecific Exception.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import random+import string+import subprocess++from googleapiclient import discovery+from googleapiclient import errors++from . import gcp+++def deploy_job(region, image_uri, chief_config, worker_count, worker_config,

Please add a docstring

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import json+import logging+import os+import random+import string+import sys+import tempfile++from . import machine_config++from docker import APIClient+from tensorflow.python.framework.versions import VERSION+++logger = logging.getLogger(__name__)+logging.basicConfig(level=logging.INFO)+++def get_file_map(entry_point, startup_script, chief_config,+                 requirements_txt=None, dst_dir='/app/',+                 docker_base_image=None):+    location_map = {}+    # Map entry_point directory to the dst directory.+    entry_point_dir, _ = os.path.split(entry_point)+    if entry_point_dir == '':  # Current directory+        entry_point_dir = '.'+    location_map[entry_point_dir] = dst_dir++    # Place startup_script in the dst directory.+    _, startup_file_name = os.path.split(startup_script)+    location_map[startup_script] = os.path.join(dst_dir, startup_file_name)++    # Place requirements_txt in the dst directory.+    if requirements_txt is not None:+        _, requirements_txt_name = os.path.split(+            requirements_txt)+        location_map[requirements_txt] = os.path.join(+            dst_dir, requirements_txt_name)++    # Place docker file in the root directory.+    docker_file = _create_docker_file(+        startup_script, chief_config, requirements_txt,+        dst_dir, docker_base_image)+    location_map[docker_file] = 'Dockerfile'+    return docker_file, location_map+++def get_docker_image(docker_registry, tar_file):+    docker_client = APIClient(version='auto')+    # create docker image from tarball+    image_tag = _build_docker_image(docker_registry, tar_file, docker_client)+    # push to the registry+    _publish_docker_image(image_tag, docker_client)+    return image_tag+++def _build_docker_image(docker_registry, tar_file, docker_client):+    image_tag = _generate_name(docker_registry)+    logger.info(' Building docker image: {}'.format(image_tag))+    with open(tar_file, 'rb') as fileobj:+        bld_logs_generator = docker_client.build(+            path='.',+            custom_context=True,+            fileobj=fileobj,+            tag=image_tag,+            encoding='utf-8')+    _get_logs(bld_logs_generator, 'build')+    return image_tag+++def _publish_docker_image(image_tag, docker_client):+    logger.info(' Publishing docker image: {}'.format(image_tag))+    pb_logs_generator = docker_client.push(image_tag, stream=True)+    _get_logs(pb_logs_generator, 'publish')+++def _create_docker_file(startup_script, chief_config, requirements_txt,+                        dst_dir, docker_base_image):+    # Create a Dockerfile.+    _, output_file = tempfile.mkstemp()++    if docker_base_image is None:+        # Get the TF docker base image to use based on the current TF+        # and python version.+        docker_base_image = 'tensorflow/tensorflow:{}'.format(VERSION)+        if (chief_config.accelerator_type !=+                machine_config.AcceleratorType.NO_ACCELERATOR):+            docker_base_image += '-gpu'++        if sys.version_info[0] == 3:+            docker_base_image += '-py3'++    lines = ['FROM {}'.format(docker_base_image), 'WORKDIR {}'.format(dst_dir)]+    lines.append('COPY {} {}'.format(dst_dir, dst_dir))++    if requirements_txt is not None:+        _, requirements_txt_name = os.path.split(requirements_txt)+        dst_requirements_txt = os.path.join(requirements_txt_name)+        # install pip requirements from requirements_txt if it exists.+        lines.append('RUN if [ -e {} ]; '+                     'then pip install --no-cache -r {}; '+                     'fi'.format(dst_requirements_txt, dst_requirements_txt))++    _, startup_file_name = os.path.split(startup_script)+    # Using `ENTRYPOINT` here instead of `CMD` specifically because we want to+    # support passing user code flags.+    lines.extend([+        'ENTRYPOINT ["python", "{}"]'.format(startup_file_name)+    ])++    content = '\n'.join(lines)+    with open(output_file, 'w') as f:+        f.write(content)+    return output_file+++def _generate_name(docker_registry):+    unique_tag = ''.join(random.choice(+        string.ascii_lowercase + string.digits) for _ in range(32))

This is only unique assuming a unique seed. If the user does random.seed(123) at the start of their script (which is not uncommon in ML; many Keras example scripts do this) the tag will not be unique. Consider using a hash of time.time() or something similar.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import json+import logging+import os+import random+import string+import sys+import tempfile++from . import machine_config++from docker import APIClient+from tensorflow.python.framework.versions import VERSION+++logger = logging.getLogger(__name__)+logging.basicConfig(level=logging.INFO)+++def get_file_map(entry_point, startup_script, chief_config,+                 requirements_txt=None, dst_dir='/app/',+                 docker_base_image=None):+    location_map = {}+    # Map entry_point directory to the dst directory.+    entry_point_dir, _ = os.path.split(entry_point)+    if entry_point_dir == '':  # Current directory+        entry_point_dir = '.'+    location_map[entry_point_dir] = dst_dir++    # Place startup_script in the dst directory.+    _, startup_file_name = os.path.split(startup_script)+    location_map[startup_script] = os.path.join(dst_dir, startup_file_name)++    # Place requirements_txt in the dst directory.+    if requirements_txt is not None:+        _, requirements_txt_name = os.path.split(+            requirements_txt)+        location_map[requirements_txt] = os.path.join(+            dst_dir, requirements_txt_name)++    # Place docker file in the root directory.+    docker_file = _create_docker_file(+        startup_script, chief_config, requirements_txt,+        dst_dir, docker_base_image)+    location_map[docker_file] = 'Dockerfile'+    return docker_file, location_map+++def get_docker_image(docker_registry, tar_file):+    docker_client = APIClient(version='auto')+    # create docker image from tarball+    image_tag = _build_docker_image(docker_registry, tar_file, docker_client)+    # push to the registry+    _publish_docker_image(image_tag, docker_client)+    return image_tag+++def _build_docker_image(docker_registry, tar_file, docker_client):+    image_tag = _generate_name(docker_registry)+    logger.info(' Building docker image: {}'.format(image_tag))+    with open(tar_file, 'rb') as fileobj:+        bld_logs_generator = docker_client.build(+            path='.',+            custom_context=True,+            fileobj=fileobj,+            tag=image_tag,+            encoding='utf-8')+    _get_logs(bld_logs_generator, 'build')+    return image_tag+++def _publish_docker_image(image_tag, docker_client):+    logger.info(' Publishing docker image: {}'.format(image_tag))+    pb_logs_generator = docker_client.push(image_tag, stream=True)+    _get_logs(pb_logs_generator, 'publish')+++def _create_docker_file(startup_script, chief_config, requirements_txt,+                        dst_dir, docker_base_image):+    # Create a Dockerfile.+    _, output_file = tempfile.mkstemp()++    if docker_base_image is None:+        # Get the TF docker base image to use based on the current TF+        # and python version.+        docker_base_image = 'tensorflow/tensorflow:{}'.format(VERSION)+        if (chief_config.accelerator_type !=+                machine_config.AcceleratorType.NO_ACCELERATOR):+            docker_base_image += '-gpu'++        if sys.version_info[0] == 3:

"As TensorFlow no longer publishes new Python 2 packages, we are deprecating our -py3 tagging behavior for container images. Previously, we published Python 2 images without named tag annotations, and Python 3 images with -py3 tag annotations."

It should not be necessary to support Py2, no?

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import json+import logging+import os+import random+import string+import sys+import tempfile++from . import machine_config++from docker import APIClient+from tensorflow.python.framework.versions import VERSION+++logger = logging.getLogger(__name__)+logging.basicConfig(level=logging.INFO)+++def get_file_map(entry_point, startup_script, chief_config,+                 requirements_txt=None, dst_dir='/app/',+                 docker_base_image=None):+    location_map = {}+    # Map entry_point directory to the dst directory.+    entry_point_dir, _ = os.path.split(entry_point)+    if entry_point_dir == '':  # Current directory+        entry_point_dir = '.'+    location_map[entry_point_dir] = dst_dir++    # Place startup_script in the dst directory.+    _, startup_file_name = os.path.split(startup_script)+    location_map[startup_script] = os.path.join(dst_dir, startup_file_name)++    # Place requirements_txt in the dst directory.+    if requirements_txt is not None:+        _, requirements_txt_name = os.path.split(+            requirements_txt)+        location_map[requirements_txt] = os.path.join(+            dst_dir, requirements_txt_name)++    # Place docker file in the root directory.+    docker_file = _create_docker_file(+        startup_script, chief_config, requirements_txt,+        dst_dir, docker_base_image)+    location_map[docker_file] = 'Dockerfile'+    return docker_file, location_map+++def get_docker_image(docker_registry, tar_file):+    docker_client = APIClient(version='auto')+    # create docker image from tarball+    image_tag = _build_docker_image(docker_registry, tar_file, docker_client)+    # push to the registry+    _publish_docker_image(image_tag, docker_client)+    return image_tag+++def _build_docker_image(docker_registry, tar_file, docker_client):

Q: is it the case that all functions that aren't prefixed with _ are meant to be publicly exposed?

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import json+import logging+import os+import random+import string+import sys+import tempfile++from . import machine_config++from docker import APIClient+from tensorflow.python.framework.versions import VERSION+++logger = logging.getLogger(__name__)+logging.basicConfig(level=logging.INFO)+++def get_file_map(entry_point, startup_script, chief_config,+                 requirements_txt=None, dst_dir='/app/',+                 docker_base_image=None):+    location_map = {}+    # Map entry_point directory to the dst directory.+    entry_point_dir, _ = os.path.split(entry_point)+    if entry_point_dir == '':  # Current directory+        entry_point_dir = '.'+    location_map[entry_point_dir] = dst_dir++    # Place startup_script in the dst directory.+    _, startup_file_name = os.path.split(startup_script)+    location_map[startup_script] = os.path.join(dst_dir, startup_file_name)++    # Place requirements_txt in the dst directory.+    if requirements_txt is not None:+        _, requirements_txt_name = os.path.split(+            requirements_txt)+        location_map[requirements_txt] = os.path.join(+            dst_dir, requirements_txt_name)++    # Place docker file in the root directory.+    docker_file = _create_docker_file(+        startup_script, chief_config, requirements_txt,+        dst_dir, docker_base_image)+    location_map[docker_file] = 'Dockerfile'+    return docker_file, location_map+++def get_docker_image(docker_registry, tar_file):+    docker_client = APIClient(version='auto')+    # create docker image from tarball+    image_tag = _build_docker_image(docker_registry, tar_file, docker_client)+    # push to the registry+    _publish_docker_image(image_tag, docker_client)+    return image_tag+++def _build_docker_image(docker_registry, tar_file, docker_client):+    image_tag = _generate_name(docker_registry)+    logger.info(' Building docker image: {}'.format(image_tag))+    with open(tar_file, 'rb') as fileobj:+        bld_logs_generator = docker_client.build(+            path='.',+            custom_context=True,+            fileobj=fileobj,+            tag=image_tag,+            encoding='utf-8')+    _get_logs(bld_logs_generator, 'build')+    return image_tag+++def _publish_docker_image(image_tag, docker_client):+    logger.info(' Publishing docker image: {}'.format(image_tag))+    pb_logs_generator = docker_client.push(image_tag, stream=True)+    _get_logs(pb_logs_generator, 'publish')+++def _create_docker_file(startup_script, chief_config, requirements_txt,

This also deserves a docstring.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import json+import logging+import os+import random+import string+import sys+import tempfile++from . import machine_config++from docker import APIClient+from tensorflow.python.framework.versions import VERSION+++logger = logging.getLogger(__name__)+logging.basicConfig(level=logging.INFO)+++def get_file_map(entry_point, startup_script, chief_config,+                 requirements_txt=None, dst_dir='/app/',+                 docker_base_image=None):+    location_map = {}+    # Map entry_point directory to the dst directory.+    entry_point_dir, _ = os.path.split(entry_point)+    if entry_point_dir == '':  # Current directory+        entry_point_dir = '.'+    location_map[entry_point_dir] = dst_dir++    # Place startup_script in the dst directory.+    _, startup_file_name = os.path.split(startup_script)+    location_map[startup_script] = os.path.join(dst_dir, startup_file_name)++    # Place requirements_txt in the dst directory.+    if requirements_txt is not None:+        _, requirements_txt_name = os.path.split(+            requirements_txt)+        location_map[requirements_txt] = os.path.join(+            dst_dir, requirements_txt_name)++    # Place docker file in the root directory.+    docker_file = _create_docker_file(+        startup_script, chief_config, requirements_txt,+        dst_dir, docker_base_image)+    location_map[docker_file] = 'Dockerfile'+    return docker_file, location_map+++def get_docker_image(docker_registry, tar_file):

I believe it's a filepath, not a file. The name is ambiguous (same in _build_docker_image).

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import json+import logging+import os+import random+import string+import sys+import tempfile++from . import machine_config++from docker import APIClient+from tensorflow.python.framework.versions import VERSION+++logger = logging.getLogger(__name__)+logging.basicConfig(level=logging.INFO)+++def get_file_map(entry_point, startup_script, chief_config,

Please add a docstring.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import

Please add a file docstring describing in 1-2 sentences what this file contains and how it relates to the general architecture of the project. Please do this for each of the main code files.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import json+import logging+import os+import random+import string+import sys+import tempfile++from . import machine_config++from docker import APIClient+from tensorflow.python.framework.versions import VERSION+++logger = logging.getLogger(__name__)+logging.basicConfig(level=logging.INFO)+++def get_file_map(entry_point, startup_script, chief_config,+                 requirements_txt=None, dst_dir='/app/',

In general, avoid using abbreviation in arguments unless they're very common and immediately clear, e.g. prefer "destination_dir" (or just "destination" if it is sufficiently clear that it is a string path to a directory). "dir" is ok since it's very common and not ambiguous.

Same in _create_docker_file.

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

  <h2>What is this repo?</h2> -This repository provides APIs that will allow you to easily go from debugging and training your TensorFlow code in a local environment to distributed training in the cloud.+This repository provides APIs that will allow to easily go from debugging and training your TensorFlow code in a local environment to distributed training in the cloud.

"your Keras and TensorFlow code"

pavithrasv

comment created time in 17 days

Pull request review commenttensorflow/cloud

Initial commit.

  <h2>What is this repo?</h2> -This repository provides APIs that will allow you to easily go from debugging and training your TensorFlow code in a local environment to distributed training in the cloud.+This repository provides APIs that will allow to easily go from debugging and training your TensorFlow code in a local environment to distributed training in the cloud.+++## Usage++Usage with `tf.keras` script that trains using `model.fit` [tests/call_run_on_script_with_keras_fit.py](tests/call_run_on_script_with_keras_fit.py).

Please use bullet point formatting. You can put the link directly on "Usage with tf.keras script that trains using" (displaying the filename is not very useful) -- same below

pavithrasv

comment created time in 17 days

PR opened qlzh727/community

Proofreading, etc.

Only small nits :)

+160 -132

0 comment

1 changed file

pr created time in 21 days

push eventfchollet/community

François Chollet

commit sha 0405d803912771e21eec0590cb70fb9d63fdc421

Update 20200205-standalone-keras-repository.md

view details

push time in 21 days

Pull request review commentkeras-team/autokeras

Adopting the tf.keras preprocessing layers

+import kerastuner+import tensorflow as tf+from tensorflow.python.util import nest++from autokeras import utils+from autokeras.engine import node as node_module+from autokeras.engine import serializable+++class Block(kerastuner.HyperModel, serializable.Serializable):+    """The base class for different Block.++    The Block can be connected together to build the search space+    for an AutoModel. Notably, many args in the __init__ function are defaults to+    be a tunable variable when not specified by the user.++    # Arguments+        name: String. The name of the block. If unspecified, it will be set+        automatically with the class name.

Need indent

haifeng-jin

comment created time in a month

Pull request review commentkeras-team/autokeras

Adopting the tf.keras preprocessing layers

+import numpy as np+from tensorflow.keras.layers.experimental import preprocessing+from tensorflow.python.util import nest++from autokeras import keras_layers+from autokeras.adapters import input_adapter+from autokeras.engine import block as block_module+++class Normalization(block_module.Block):+    """ Perform basic image transformation and augmentation.++    # Arguments+        axis: Integer or tuple of integers, the axis or axes that should be+            normalized (typically the features axis). We will normalize each element+            in the specified axis. The default is '-1' (the innermost axis); 0 (the+            batch axis) is not allowed.+    """++    def __init__(self, axis=-1, **kwargs):+        super().__init__(**kwargs)+        self.axis = axis++    def build(self, hp, inputs=None):+        input_node = nest.flatten(inputs)[0]+        return preprocessing.Normalization(axis=self.axis)(input_node)++    def get_config(self):+        config = super().get_config()+        config.update({'axis': self.axis})+        return config+++class TextToIntSequence(block_module.Block):+    """Convert raw texts to sequences of word indices.++    # Arguments+        output_sequence_length: Int. The maximum length of a sentence. If+            unspecified, it would be tuned automatically.+        max_tokens: Int. The maximum size of the vocabulary. Defaults to 20000.+    """++    def __init__(self,+                 output_sequence_length=None,+                 max_tokens=20000,+                 **kwargs):+        super().__init__(**kwargs)+        self.output_sequence_length = output_sequence_length+        self.max_tokens = max_tokens++    def get_config(self):+        config = super().get_config()+        config.update({+            'max_len': self.max_len,+            'max_tokens': self.vocab_size,+        })+        return config++    def build(self, hp, inputs=None):+        input_node = nest.flatten(inputs)[0]+        if self.output_sequence_length is not None:+            output_sequence_length = self.output_sequence_length+        else:+            output_sequence_length = hp.Choice('output_sequence_length',+                                               [64, 128, 256, 512], default=64)+        output_node = preprocessing.TextVectorization(+            max_tokens=self.max_tokens,+            output_mode='int',+            output_sequence_length=output_sequence_length)(input_node)+        return output_node+++class TextToNgramVector(block_module.Block):+    """Convert raw texts to n-gram vectors.++    # Arguments+        max_tokens: Int. The maximum size of the vocabulary. Defaults to 20000.+    """++    def __init__(self,+                 max_tokens=20000,+                 **kwargs):+        super().__init__(**kwargs)+        self.max_tokens = max_tokens++    def build(self, hp, inputs=None):+        input_node = nest.flatten(inputs)[0]+        return preprocessing.TextVectorization(+            max_tokens=self.max_tokens,+            output_mode='tf-idf')(input_node)++    def get_config(self):+        config = super().get_config()+        config.update({'max_tokens': self.max_tokens})+        return config+++class ImageAugmentation(block_module.Block):+    """Collection of various image augmentation methods.++    # Arguments+        percentage: Float. The percentage of data to augment.+        rotation_range: Int. The value can only be 0, 90, or 180.+            Degree range for random rotations. Default to 180.+        random_crop: Boolean. Whether to crop the image randomly. Default to True.+        brightness_range: Positive float.+            Serve as 'max_delta' in tf.image.random_brightness. Default to 0.5.+            Equivalent to adjust brightness using a 'delta' randomly picked in+            the interval [-max_delta, max_delta).+        saturation_range: Positive float or Tuple.+            If given a positive float, _get_min_and_max() will automated generate+            a tuple for saturation range. If given a tuple directly, it will serve+            as a range for picking a saturation shift value from. Default to 0.5.+        contrast_range: Positive float or Tuple.+            If given a positive float, _get_min_and_max() will automated generate+            a tuple for contrast range. If given a tuple directly, it will serve+            as a range for picking a contrast shift value from. Default to 0.5.+        translation: Boolean. Whether to translate the image.+        horizontal_flip: Boolean. Whether to flip the image horizontally.+        vertical_flip: Boolean. Whether to flip the image vertically.+        gaussian_noise: Boolean. Whether to add gaussian noise to the image.+    """++    def __init__(self,+                 percentage=0.25,+                 rotation_range=180,+                 random_crop=True,+                 brightness_range=0.5,+                 saturation_range=0.5,+                 contrast_range=0.5,+                 translation=True,+                 horizontal_flip=True,+                 vertical_flip=True,+                 gaussian_noise=True,+                 **kwargs):+        super().__init__(**kwargs)+        self.percentage = percentage+        self.rotation_range = rotation_range+        self._rotate_choices = [0]+        if self.rotation_range == 90:+            self._rotate_choices = [0, 1, 3]+        elif self.rotation_range == 180:+            self._rotate_choices = [0, 1, 2, 3]+        self.random_crop = random_crop+        if self.random_crop:+            # Generate 20 crop settings, ranging from a 1% to 20% crop.+            self.scales = list(np.arange(0.8, 1.0, 0.01))+            self.boxes = np.zeros((len(self.scales), 4))+            for i, scale in enumerate(self.scales):+                x1 = y1 = 0.5 - (0.5 * scale)+                x2 = y2 = 0.5 + (0.5 * scale)+                self.boxes[i] = [x1, y1, x2, y2]+        self.brightness_range = brightness_range+        self.saturation_range = self._get_min_and_max(saturation_range,+                                                      'saturation_range')+        self.contrast_range = self._get_min_and_max(contrast_range,+                                                    'contrast_range')+        self.translation = translation+        self.horizontal_flip = horizontal_flip+        self.vertical_flip = vertical_flip+        self.gaussian_noise = gaussian_noise+        self.shape = None++    @staticmethod+    def _get_min_and_max(value, name):+        if isinstance(value, (tuple, list)) and len(value) == 2:+            min_value, max_value = value+            return min_value, max_value+        elif isinstance(value, (int, float)):+            min_value = 1. - value+            max_value = 1. + value+            return min_value, max_value+        elif value == 0:+            return None+        else:+            raise ValueError('Expected {name} to be either a float between 0 and 1, '+                             'or a tuple of 2 floats between 0 and 1, '+                             'but got {value}'.format(name=name, value=value))++    def build(self, hp, inputs=None):+        return inputs+++class FeatureEncoding(block_module.Block):

"FeatureEncoding" is very generic (everything in ML is a feature and needs encoding, since it usually doesn't start out as a numerical array). Perhaps CategoricalEncoding or CategoricalFeatureEncoding would be more specific? Or CategoricalToNumerical?

haifeng-jin

comment created time in a month

Pull request review commentkeras-team/autokeras

Adopting the tf.keras preprocessing layers

+import numpy as np+from tensorflow.keras.layers.experimental import preprocessing+from tensorflow.python.util import nest++from autokeras import keras_layers+from autokeras.adapters import input_adapter+from autokeras.engine import block as block_module+++class Normalization(block_module.Block):

Note that there is a Normalization layer in Keras.

haifeng-jin

comment created time in a month

Pull request review commentkeras-team/autokeras

Adopting the tf.keras preprocessing layers

 def _check(self, x):             raise TypeError('Expect the data to TextInput to be strings, but got '                             '{type}.'.format(type=x.dtype)) -    def _convert_to_dataset(self, x):+    def convert_to_dataset(self, x):+        if len(x.shape) == 1:+            x = x.reshape(-1, 1)         if isinstance(x, np.ndarray):             x = tf.data.Dataset.from_tensor_slices(x)         return x  -class StructuredDataInput(Input):-    """Input node for structured data.--    The input data should be numpy.ndarray, pandas.DataFrame or tensorflow.Dataset.-    The data should be two-dimensional with numerical or categorical values.+class StructuredDataInputAdapter(adapter_module.Adapter): -    # Arguments-        column_names: A list of strings specifying the names of the columns. The-            length of the list should be equal to the number of columns of the data.-            Defaults to None. If None, it will be obtained from the header of the csv-            file or the pandas.DataFrame.-        column_types: Dict. The keys are the column names. The values should either-            be 'numerical' or 'categorical', indicating the type of that column.-            Defaults to None. If not None, the column_names need to be specified.-            If None, it will be inferred from the data. A column will be judged as-            categorical if the number of different values is less than 5% of the-            number of instances.-    """+    CATEGORICAL = 'categorical'+    NUMERICAL = 'numerical'

I would recommend not having global keys like this be class attributes. Instead, make it external constants.

haifeng-jin

comment created time in a month

issue commentkeras-team/autokeras

Customize the optimizer

Do we ever expect users to want to specify losses and metrics otherwise than through the Head objects?

We could add a compile step with only the optimizer argument.

haifeng-jin

comment created time in a month

pull request commentkeras-team/autokeras

Adding some types hints.

However, since I am really not familiar with this mechanism there might be some cons we are not aware of.

What are some potential issues?

gabrieldemarmiesse

comment created time in a month

Pull request review commentkeras-team/autokeras

Adding some types hints.

 class ImageClassifier(SupervisedImagePipeline):     """      def __init__(self,-                 num_classes=None,-                 multi_label=False,-                 loss=None,-                 metrics=None,-                 name='image_classifier',-                 max_trials=100,-                 directory=None,-                 objective='val_loss',-                 overwrite=True,-                 seed=None):+                 num_classes: Optional[int] = None,+                 multi_label: bool = False,+                 loss: Union[str, Callable, None] = None,+                 metrics: Union[str, Callable, None] = None,

This is actually a list.

gabrieldemarmiesse

comment created time in a month

pull request commentkeras-team/governance

RFC: Keras Grouped Convolution

I think it is reasonable to add this to the core tf.keras API.

One important thing is that when in a configuration where grouping is not supported, the error message raised should be readable and actionable.

What's your suggested implementation at the level of the base Conv layer?

lgeiger

comment created time in 2 months

Pull request review commentkeras-team/autokeras

Add the export model API

 def evaluate(self, x, y=None, batch_size=32, **kwargs):         data = preprocess_graph.preprocess(             self._process_xy(x, y))[0].batch(batch_size)         return model.evaluate(data, **kwargs)++    def export_keras_model(self):

Would export_model suffice here?

haifeng-jin

comment created time in 2 months

Pull request review commentkeras-team/autokeras

Add the export model API

 def evaluate(self, x, y=None, batch_size=32, **kwargs):         data = preprocess_graph.preprocess(             self._process_xy(x, y))[0].batch(batch_size)         return model.evaluate(data, **kwargs)++    def export_keras_model(self):+        """Export the best Keras Model.++        # Returns+            tf.keras.Model. The best model found during the search with trained

"tf.keras.Model instance. The best model found during the search, loaded with trained weights."

haifeng-jin

comment created time in 2 months

push eventfchollet/ARC

James McDermott

commit sha 79192375e55642199d77faec785bc76dbe143fdd

Distinguish cell border colour from grey cell colour #AAAAAA.

view details

François Chollet

commit sha 1f68da7cf7c5b1849cef67f0e2d74680b42306a8

Merge pull request #25 from jmmcd/master Distinguish cell border colour from grey cell colour #AAAAAA.

view details

push time in 2 months

PR merged fchollet/ARC

Distinguish cell border colour from grey cell colour #AAAAAA.

To avoid grey cells like this.

grey_cells

+2 -2

2 comments

1 changed file

jmmcd

pr closed time in 2 months

pull request commentfchollet/ARC

Distinguish cell border colour from grey cell colour #AAAAAA.

What does the new version look like?

jmmcd

comment created time in 2 months

create barnchtensorflow/tensorflow

branch : edit_release_notes

created branch time in 2 months

Pull request review commentkeras-team/autokeras

Overwrite

 def __init__(self,                  directory=None,                  objective='val_loss',                  tuner='greedy',+                 overwrite=True,

It defaults to False in Keras Tuner. Having a different default here would be confusing.

jhfjhfj1

comment created time in 3 months

PR opened tensorflow/tensorflow

Unify V1/2 layer naming in internal imports

This fixes V2 behavior issues in places where we import keras.layers internally (e.g. in Keras Applications).

+58 -21

0 comment

5 changed files

pr created time in 2 months

create barnchtensorflow/tensorflow

branch : layer_imports_internal

created branch time in 2 months

pull request commentkeras-team/keras-applications

Fix symbolic bias_add cast

What's the nature of the fix here?

lagejoao

comment created time in 3 months

push eventfchollet/ARC

Aurelien Geron

commit sha ff80b972251c4cb47475e901b1805ce2274f896a

Fixes #21 by clearing file input on click or random task loaded

view details

François Chollet

commit sha eccfbf59f11a7537cee9773771b86b4118b4df17

Merge pull request #22 from ageron/fix_file_reload Fixes #21 by clearing file input on click or random task loaded

view details

push time in 3 months

issue closedfchollet/ARC

Choose File not responding

The ARC testing interface does not load a file if it was loaded before selecting a random task.

To replicate, perform the following twice:

  • Click "Random task"
  • Click "Choose File" and select a file (must be same file both times)

The last step should load the selected file, but instead does nothing.

closed time in 3 months

jamesmckownbishop

PR closed fchollet/ARC

Initial setup
+64 -60

1 comment

5 changed files

kbines

pr closed time in 3 months

pull request commentfchollet/ARC

Initial setup

This PR was apparently opened by mistake. Closing.

kbines

comment created time in 3 months

push eventfchollet/ARC

Paul Willot

commit sha 3150c93a47ca37635bbd989e2cc636b3b5d3b160

Fix task demonstration for 6a1e5592.json

view details

Paul Willot

commit sha 79213ceb65292b0bfa626313f51a9fce9799a11f

Add testing notes

view details

m3at

commit sha 800efc866fedcb39fd611e943560fa69042b4dcb

Remove notes added by mistake

view details

François Chollet

commit sha bd9e2c934c83d00251b7b4781ffc38cd167c885f

Merge pull request #16 from m3at/master Fix task demonstration for 6a1e5592.json

view details

push time in 3 months

PR merged fchollet/ARC

Fix task demonstration for 6a1e5592.json

Hi François thanks for working on ARC it is a very valuable benchmark and insightful paper!

While playing with it I found what I believe to be a copy and paste issue on task 6a1e5592.

I understand the task as "move the grey blocks into the fitting slots, and color them blue". However the first demonstration leave the grey blocks behind. It seems confirmed by trying both answers:

Wrong: wrong_answer

Correct: correct_answer

My understanding is that this kind of ambiguity in answers are not a goal, but if I'm mistaken please ignore this PR.

+1 -1

1 comment

1 changed file

m3at

pr closed time in 3 months

pull request commentfchollet/ARC

Fix task demonstration for 6a1e5592.json

That's right, thank you for catching this!

m3at

comment created time in 3 months

issue commentfchollet/ARC

Choose File not responding

@ageron maybe this is related to the recent changes?

jamesmckownbishop

comment created time in 3 months

Pull request review commentkeras-team/autokeras

Task specific tuners

 class GraphAutoModel(AutoModel):             AutoModel in the current directory.         objective: String. Name of model metric to minimize             or maximize, e.g. 'val_accuracy'. Defaults to 'val_loss'.+        tuner: String. The tuner to be used for the search.

Likewise.

jhfjhfj1

comment created time in 3 months

Pull request review commentkeras-team/autokeras

Task specific tuners

 class RandomSearch(AutoTuner, kerastuner.RandomSearch): class HyperBand(AutoTuner, kerastuner.Hyperband):     """KerasTuner Hyperband with preprocessing layer tuning."""     pass+++class GreedyOracle(kerastuner.Oracle):+    """An oracle combining random search and greedy algorithm.++    It groups the HyperParameters into several categories, namely, HyperGraph,+    Preprocessor, Architecture, and Optimization. The oracle tunes each group+    separately using random search. In each trial, it use a greedy strategy to+    generate new values for one of the categories of HyperParameters and use the best+    trial so far for the rest of the HyperParameters values.++    # Arguments+        hyper_graph: HyperGraph. The hyper_graph model to be tuned.+        seed: Int. Random seed.+    """++    HYPER = 'HYPER'+    PREPROCESS = 'PREPROCESS'+    OPT = 'OPT'+    ARCH = 'ARCH'+    STAGES = [HYPER, PREPROCESS, OPT, ARCH]++    @staticmethod+    def next_stage(stage):+        stages = GreedyOracle.STAGES+        return stages[(stages.index(stage) + 1) % len(stages)]++    def __init__(self, hyper_graph, seed=None, **kwargs):+        super().__init__(**kwargs)+        self.hyper_graph = hyper_graph+        # Start from tuning the hyper block hps.+        self._stage = GreedyOracle.HYPER+        # Sets of HyperParameter names.+        self._hp_names = {+            GreedyOracle.HYPER: set(),+            GreedyOracle.PREPROCESS: set(),+            GreedyOracle.OPT: set(),+            GreedyOracle.ARCH: set(),+        }+        # The quota used to tune each category of hps.+        self._capacity = {+            GreedyOracle.HYPER: 1,+            GreedyOracle.PREPROCESS: 1,+            GreedyOracle.OPT: 1,+            GreedyOracle.ARCH: 4,+        }+        self._stage_trial_count = 0+        self.seed = seed or random.randint(1, 1e4)+        # Incremented at every call to `populate_space`.+        self._seed_state = self.seed+        self._tried_so_far = set()+        self._max_collisions = 5++    def set_state(self, state):+        super().set_state(state)+        # TODO: self.hyper_graph.set_state(state['hyper_graph'])+        # currently the state is not json serializable.+        self._stage = state['stage']+        self._capacity = state['capacity']++    def get_state(self):+        state = super().get_state()+        state.update({+            # TODO: 'hyper_graph': self.hyper_graph.get_state(),+            # currently the state is not json serializable.+            'stage': self._stage,+            'capacity': self._capacity,+        })+        return state++    def update_space(self, hyperparameters):+        # Get the block names.+        preprocess_graph, keras_graph = self.hyper_graph.build_graphs(+            hyperparameters)++        # Add the new Hyperparameters to different categories.+        ref_names = {hp.name for hp in self.hyperparameters.space}+        for hp in hyperparameters.space:+            if hp.name not in ref_names:+                hp_type = None+                if any([hp.name.startswith(block.name)+                        for block in self.hyper_graph.blocks+                        if isinstance(block, base.HyperBlock)]):+                    hp_type = GreedyOracle.HYPER+                elif any([hp.name.startswith(block.name)+                          for block in preprocess_graph.blocks]):+                    hp_type = GreedyOracle.PREPROCESS+                elif any([hp.name.startswith(block.name)+                          for block in keras_graph.blocks]):+                    hp_type = GreedyOracle.ARCH+                else:+                    hp_type = GreedyOracle.OPT+                self._hp_names[hp_type].add(hp.name)++        super().update_space(hyperparameters)++    def _populate_space(self, trial_id):+        for _ in range(len(GreedyOracle.STAGES)):+            values = self._generate_stage_values()+            # Reached max collisions.+            if values is None:+                # Try next stage.+                self._stage = GreedyOracle.next_stage(self._stage)+                self._stage_trial_count = 0+                continue+            # Values found.+            self._stage_trial_count += 1+            if self._stage_trial_count == self._capacity[self._stage]:+                self._stage = GreedyOracle.next_stage(self._stage)+                self._stage_trial_count = 0+            return {'status': kerastuner.engine.trial.TrialStatus.RUNNING,+                    'values': values}+        # All stages reached max collisions.+        return {'status': kerastuner.engine.trial.TrialStatus.STOPPED,+                'values': None}++    def _generate_stage_values(self):+        best_trials = self.get_best_trials()+        if best_trials:+            best_values = best_trials[0].hyperparameters.values+        else:+            best_values = self.hyperparameters.values+        collisions = 0+        while 1:+            # Generate new values for the current stage.+            values = {}+            for p in self.hyperparameters.space:+                if p.name in self._hp_names[self._stage]:+                    values[p.name] = p.random_sample(self._seed_state)+                    self._seed_state += 1+            values = {**best_values, **values}+            # Keep trying until the set of values is unique,+            # or until we exit due to too many collisions.+            values_hash = self._compute_values_hash(values)+            if values_hash not in self._tried_so_far:+                self._tried_so_far.add(values_hash)+                break+            collisions += 1+            if collisions > self._max_collisions:+                # Reached max collisions. No value to return.+                return None+        return values+++class Greedy(AutoTuner):++    def __init__(self,+                 hyper_graph,+                 hypermodel,+                 objective,+                 max_trials,+                 fit_on_val_data=False,+                 seed=None,+                 hyperparameters=None,+                 tune_new_entries=True,+                 allow_new_entries=True,+                 **kwargs):+        self.seed = seed+        oracle = GreedyOracle(+            hyper_graph=hyper_graph,+            objective=objective,+            max_trials=max_trials,+            seed=seed,+            hyperparameters=hyperparameters,+            tune_new_entries=tune_new_entries,+            allow_new_entries=allow_new_entries)+        hp = oracle.get_space()+        preprocess_graph, keras_graph = hyper_graph.build_graphs(hp)+        oracle.update_space(hp)+        super().__init__(+            hyper_graph=hyper_graph,+            fit_on_val_data=fit_on_val_data,+            oracle=oracle,+            hypermodel=hypermodel,+            **kwargs)+++TUNER_CLASSES = {+    'random_search': RandomSearch,+    'greedy': Greedy,+    'image_classifier': Greedy,+    'image_regressor': Greedy,+    'text_classifier': Greedy,+    'text_regressor': Greedy,+    'structured_data_classifier': Greedy,+    'structured_data_regressor': Greedy,+}+++def get_tuner_class(name):

This needs to be extended to accept passing a tuner instance rather than a string.

Also, if an invalid value is passed, the error message should be informative and actionable, saying, "the value {value} passed for argument tuner is invalid, expected one of {list of values}"

jhfjhfj1

comment created time in 3 months

Pull request review commentkeras-team/autokeras

Task specific tuners

 def __init__(self,                  max_trials=100,                  directory=None,                  objective='val_loss',+                 tuner='greedy',                  seed=None):         self.inputs = nest.flatten(inputs)         self.outputs = nest.flatten(outputs)         self.name = name-        self.tuner = None         self.max_trials = max_trials         self.directory = directory         self.seed = seed         self.hyper_graph = None         self.objective = objective+        self.tuner = tuner

Why not self.tuner = tuner_module.get_tuner_class(self.tuner)? This would be more consistent.

jhfjhfj1

comment created time in 3 months

Pull request review commentkeras-team/autokeras

Task specific tuners

 class AutoModel(object):             AutoModel in the current directory.         objective: String. Name of model metric to minimize             or maximize, e.g. 'val_accuracy'. Defaults to 'val_loss'.+        tuner: String. The tuner to be used for the search.

The accepted values and types should be listed explicitly

jhfjhfj1

comment created time in 3 months

Pull request review commentkeras-team/autokeras

Task specific tuners

 def __init__(self,                  max_trials=100,                  directory=None,                  objective='val_loss',+                 tuner='greedy',

This should also optionally accept a tuner instance (like for layer activations, initializers, etc). This makes the API more flexible (can change the tuner without having to fork the codebase)

jhfjhfj1

comment created time in 3 months

Pull request review commentkeras-team/autokeras

Epoch Speed Improvement

 class ImageAugmentation(base.Preprocessor):     """Collection of various image augmentation methods.      # Arguments+        percentage: Float. The percentage of data to augment.         rotation_range: Int. The value can only be 0, 90, or 180.             Degree range for random rotations. Default to 180.-        random_crop: Boolean. Whether to crop the image randomly. Default to True.+        crop: Boolean. Whether to crop the image randomly. Default to True.

Why change the name? crop could be interpreted in different ways: for instance, in the preprocessing layers API, we have CenterCrop and RandomCrop.

jhfjhfj1

comment created time in 3 months

pull request commentfchollet/ARC

Add random task button to load task from web

I agree with your introspective analysis. The rough steps are, 1) parse the grids into objects, 2) generate the space of all possible relationships among these objects, 3) reduce that space to the relationships that match the data, 4) apply the resulting program to the input grid. 1) and 2) can be encoded in program form via a kind of ARC DSL.

What's most interesting to me is that we're able to effortlessly identify these relationships without trying them all exhaustively (seemingly), whereas the space of possible relationships is very large. The same could be said about object parsing.

ageron

comment created time in 3 months

push eventfchollet/ARC

Aurelien Geron

commit sha 62316fa5843896f0a8fd8e5386fdf9a00f2c5843

Remove modal start and add random task button to load from web

view details

Aurelien Geron

commit sha 1135dc2b731664de036f4131bfca852eaa4aad03

Merge remote-tracking branch 'upstream/master' into web_load

view details

Aurelien Geron

commit sha 43fad5893781886296badcf8d5ff4cf3b4c57a86

Add modal back

view details

Aurelien Geron

commit sha 9fa60cfd52f481a765ec9afa6f0189d3cdd80b8a

Cleanup: clarify function name and welcome message, remove redundant welcome message, add missing return statements

view details

François Chollet

commit sha dabf02556669664b11ec0f39a71f47592f86c6aa

Merge pull request #7 from ageron/web_load Add random task button to load task from web

view details

push time in 3 months

PR merged fchollet/ARC

Add random task button to load task from web

This PR makes it easier for people to get started quickly with the tasks without having to load a new task all the time. Plus, it makes it easier to host this UI on a website so people don't have to download all the JSON tasks locally. I could have done a "next task" and "previous task" button, but when playing around with it I found that I liked having a random challenge.

+62 -33

3 comments

3 changed files

ageron

pr closed time in 3 months

pull request commentfchollet/ARC

Remove modal start and add random task button to load from web

Thank you for the PR!

I would recommend keeping the modal and adding a "Pick task at random" button to the modal.

ageron

comment created time in 3 months

PR closed fchollet/ARC

Visualize training and test set

Given the variety of examples, I find it helpful to create a single HTML page to visualize the complete set of training and test examples.

This PR consists of a simple python script that iterates through data/training/*.json, generates HTML and injects into a template.

That is, run

python visualize_training_set.py

and it uses the template training_template.html to generate output file training_data.html

Screen Shot 2019-11-17 at 4 16 39 PM

+78 -0

1 comment

2 changed files

leapingllamas

pr closed time in 3 months

pull request commentfchollet/ARC

Visualize training and test set

Thank you for the PR!

I think this is a useful tool, but I don't think it is necessary to add it to the ARC repo. The ARC repo is only meant to contain the bare minimum. I would recommend maintaining this script as part of your own fork or as part of a 3rd party "ARC tools" type repo.

leapingllamas

comment created time in 3 months

create barnchkeras-team/keras-tuner

branch : readme

created branch time in 3 months

push eventkeras-team/governance

Adam Gibson

commit sha ae885c27f20d174379ecab29f3507bd716dfab5f

Update README.md Update contact info

view details

François Chollet

commit sha 766f5e72e235fee81e687a21a2f8c7b9a7b8d6ed

Merge pull request #12 from agibsonccc/patch-1 Update README.md

view details

push time in 3 months

more