profile
viewpoint

keras-team/keras 47499

Deep Learning for humans

keras-team/keras-preprocessing 723

Utilities for working with image data, text data, and sequence data.

pavithrasv/build 0

Build-related tools for TensorFlow

pavithrasv/examples 0

TensorFlow examples

pavithrasv/fairing 0

Python SDK for building, training, and deploying ML models

pavithrasv/keras 0

Deep Learning for humans

pavithrasv/keras-tuner 0

Hyperparameter tuning for humans

issue closedtensorflow/tensorflow

[tf.keras] Model.metrics_names bug in TensorFlow 2.2.0

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Y
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Mac OS
  • TensorFlow version (use command below): 2.2.0rc0
  • Python version: 3.7

Describe the current behavior

Model.metrics_names returns an empty list (see below example) for a compiled model. This is new unexpected behavior as of TensorFlow 2.2.0 (not the case in TensorFlow 2.1.0)

These metrics names are important at compile time because they can be used to check against monitored quantities in callback. E.g. if a ModelCheckpoint callback is trying to monitor val_lss we can easily catch the typo before calling model.fit or finishing the first epoch of training.

Describe the expected behavior

Model.metric returns metrics and loss names (see below example).

Standalone code to reproduce the issue

import tensorflow as tf


inputs = tf.keras.layers.Input(shape=(3,))
outputs = tf.keras.layers.Dense(2, name="out_1")(inputs)
net = tf.keras.models.Model(inputs=inputs, outputs=outputs)
net.compile(optimizer="Adam", loss="mse", metrics=["mae"])

net.metrics_names = []  # TensorFlow 2.2.0rc2
net.metrics_names = ["loss", "mae"] # TensorFlow 2.1.0

closed time in 10 hours

briannemsick

issue commenttensorflow/tensorflow

[tf.keras] Model.metrics_names bug in TensorFlow 2.2.0

@briannemsick there are constraints because of which we will not be able to add a warning but we have updated the docs with a note and examples to clarify the behavior.

briannemsick

comment created time in 10 hours

issue commenttensorflow/tensorflow

[tf.keras] Model.metrics_names bug in TensorFlow 2.2.0

@briannemsick We are looking into adding a warning as well. Thank you.

@bjtho08 Earlier you will be able to see metrics_name after compile call for functional model but not for subclassed model.

briannemsick

comment created time in 15 hours

push eventtensorflow/cloud

Pavithra Vijay

commit sha 48dd52c0bd5ce55a3aabc3b14f36c599890547d9

Adding installation instructions to README file.

view details

Pavithra Vijay

commit sha 5ec4ea7c4df70900484a2d3863474483be6a824e

Merge pull request #9 from pavithrasv/master Adding installation instructions to README file.

view details

push time in a day

PR opened tensorflow/cloud

Reviewers
Adding installation instructions to README file.
+24 -2

0 comment

1 changed file

pr created time in a day

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 48dd52c0bd5ce55a3aabc3b14f36c599890547d9

Adding installation instructions to README file.

view details

push time in a day

Pull request review commenttensorflow/cloud

Update containerize.py

 def _get_logs(logs_generator, name):     for line in logs_generator:         try:             unicode_line = line.decode('utf-8').strip()+            logger.info(unicode_line)         except UnicodeError:-            logger.warn('Unable to decode logs.')-        line = json.loads(unicode_line)-        if line.get('error'):-            raise RuntimeError(-                'Docker image {} failed: {}'.format(-                    name, str(line.get('error'))))+            logger.warning('Unable to decode logs.')+        try:            +            line = json.loads(unicode_line)+            if line.get('error'):+                raise RuntimeError(+                    'Docker image {} failed: {}'.format(name, str(+                        line.get('error'))))+        except json.decoder.JSONDecodeError:+            pass

can we raise an error here?

gogasca

comment created time in a day

push eventtensorflow/cloud

Gonzalo Gasca Meza

commit sha 8eb724cc3e4e6a514252fea3daf9c51eaefe0cd2

Update deploy.py Disable cache_discovery=False When using google-api-python-client and oauth2client, there is this warning from the GCP cloud: ``` WARNING:googleapiclient.discovery_cache:file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth Traceback (most recent call last): File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect from google.appengine.api import memcache ModuleNotFoundError: No module named 'google.appengine' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module> from oauth2client.contrib.locked_file import LockedFile ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module> from oauth2client.locked_file import LockedFile ModuleNotFoundError: No module named 'oauth2client.locked_file' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 42, in autodetect from . import file_cache File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module> "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth" ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth ```

view details

Pavithra Vijay

commit sha 9cdf0f3a9301e5aadfe8ab715a2e1b2d82cfa53d

Merge pull request #7 from gogasca/patch-2 Update deploy.py

view details

push time in a day

PR merged tensorflow/cloud

Update deploy.py

Disable cache_discovery=False

When using google-api-python-client and oauth2client, there is this warning from the GCP cloud:

WARNING:googleapiclient.discovery_cache:file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/__init__.py", line 42, in autodetect
    from . import file_cache
  File "/Users/gogasca/Documents/Development/dpe/venv/tfe/lib/python3.7/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth"
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth
+1 -1

0 comment

1 changed file

gogasca

pr closed time in a day

push eventtensorflow/cloud

Gonzalo Gasca Meza

commit sha 55a6d462cfda3bd84909b195dd87521736d95080

Update README.md Fix link

view details

Pavithra Vijay

commit sha 813d1d98a9ccf8455d29f6df02a8af547cf56ab6

Merge pull request #5 from gogasca/patch-1 Update README.md

view details

push time in a day

PR merged tensorflow/cloud

Update README.md

Fix links

+2 -2

0 comment

1 changed file

gogasca

pr closed time in a day

issue commenttensorflow/tensorflow

[tf.keras] Model.metrics_names bug in TensorFlow 2.2.0

@briannemsick Thank you for pointing this out. This is definitely a breaking change but intended. Sorry about the lack of information on this on release notes and docs. We have fixed that now.

We refactored compile/fit quite a bit and with that we made functional and subclassed model behaviors consistent. Because of this you will be able to use some of these model properties only after your model has been trained/evaluated on actual data.

briannemsick

comment created time in 2 days

issue commenttensorflow/tensorflow

Keras connectivity metadata is not set correctly in TF 2.2.0rc0

@drasmuss What is the use case for this? We do not currently have a method as this is internal implementation detail of functional model construction.

drasmuss

comment created time in 6 days

issue closedtensorflow/tensorflow

MSE loss computes different values depending on shape

System information

  • Have I written custom code: yes
  • OS Platform and Distribution: Linux Ubuntu 19.04
  • TensorFlow version: 2.1.0
  • Python version: 3.7.5
  • CUDA/cuDNN version: 10.1
  • GPU model and memory: GTX Titan X

Describe the current behavior When MSE is calculated on two vectors, if one has an additional dimension, the result is different. For instance if one is [128] and the other is [128,1] the final value is different than what is calculated with vectors containing the same values but both [128.]. Moreover the score computed for the 4 combinations I tried ( mse([128], [128[), mse([128, 1], [128[), mse([128], [128, 1]), mse([128, 1], [128, 1]) )are all different, so I don't really trust the computation at all, so there's likely a bug somewhere.

Describe the expected behavior Either that the computation is performed correctly or that an error is raised about the inouts having different shapes. Silent errors like this are difficult to debug.

Standalone code to reproduce the issue

import numpy as np
import tensorflow as tf

targets = np.array([180.858, -53.000732, 161.95107, 135.10085, -5.4330907, -86.42976, -4.4581985, -32.90381, -153.1223, -78.94036, 190.12129, -157.32057, -8.215049, -17.959564, -21.816954, 40.21217, -50.351727, 38.70819, 52.955853, 213.77878, -142.41376, 127.22313, 164.2927, -74.497314, -74.87329, 14.303827, 164.1599, 190.37862, -63.337723, 74.058975, -70.482285, -40.203323, -47.59432, -17.782892, 70.3338, -127.87029, -12.542, -31.236902, 70.227974, -81.60634, -186.79362, -176.01068, -118.73177, -74.14537, -56.437016, 98.60682, -3.1523242, 9.694114, -11.809049, -16.225067, -4.6299715, -194.44075, -138.53864, -118.06511, -201.88641, -85.310356, 91.92171, 107.94937, -44.26706, -93.79351, -9.981134, 40.544876, 131.26842, 7.305799, -97.13315, 94.43779, 146.48007, -24.092182, 32.081444, 32.98506, 93.73731, 65.58341, 36.74394, 57.02824, -78.452866, -6.0548353, -11.639992, 114.853455, -15.473761, -24.454018, -127.82523, 68.350876, -41.449036, 39.643234, -45.420197, -0.9474962, -111.20463, -10.079266, -79.32773, -93.07437, -111.04116, -47.006187, -68.18549, 36.195724, 100.86029, -74.86413, -13.0117655, 293.18875, 39.411587, 121.270706, -142.66888, 23.961506, 81.58176, -137.42886, 31.068184, 73.448364, -90.646164, 133.64107, 88.79693, -117.37866, 54.3003, -181.60715, 100.147194, 179.99359, 24.455635, 59.38088, 135.56046, 67.400925, 151.78516, 212.14339, -202.64584, 66.06116, 1.9135226, -244.05527, -70.778275, -50.001457, -194.73297, 33.012333])
predictions = np.array([0.12198464, 0.09282801, 0.09430753, 0.06222287, 0.07448876, 0.03799684, 0.02936049, 0.03837839, 0.04432172, 0.01919999, 0.07735521, 0.04389271, 0.09087924, 0.05364547, 0.01343504, 0.04935993, 0.02090639, 0.04636865, 0.06702548, 0.09186736, 0.11273132, 0.0611049 , 0.06820674, 0.07969542, 0.02481739, 0.04868209, 0.08474196, 0.0776654 , 0.03664336, 0.04501346, 0.06626878, 0.03605408, 0.02785883, 0.01698643, 0.09615672, 0.07914701, 0.02611066, 0.0447035 , 0.08619086, 0.04838634, 0.07977191, 0.06319098, 0.04025086, 0.05129454, 0.02673621, 0.05525842, 0.0054835 , 0.04647385, 0.02476176, 0.02783814, 0.11566448, 0.08409265, 0.03792451, 0.03227585, 0.0632838 , 0.08329175, 0.04616582, 0.06513302, 0.07169756, 0.05911999, 0.05913429, 0.01704707, 0.06693612, 0.04886937, 0.02549478, 0.04468452, 0.07630262, 0.05455045, 0.06637821, 0.01789702, 0.11108026, 0.03976684, 0.0171865 , 0.13416564, 0.02845822, 0.05074854, 0.04896633, 0.05221288, 0.03563176, 0.05014472, 0.05413034, 0.0347496 , 0.0645119 , 0.04159546, 0.01868404, 0.0582131 , 0.0336203 , 0.04432501, 0.03495208, 0.02673723, 0.09592278, 0.02579375, 0.01584711, 0.02812203, 0.03840974, 0.02530819, 0.08957738, 0.14304015, 0.03345468, 0.06080145, 0.09284427, 0.04770067, 0.07064755, 0.04004309, 0.02097335, 0.08742893, 0.04389744, 0.0479476 , 0.05911161, 0.0748862 , 0.06840549, 0.0580482 , 0.05427855, 0.10075781, 0.01691986, 0.04473659, 0.0634447 , 0.03176469, 0.05624699, 0.12614223, 0.08688905, 0.02355402, 0.0871409 , 0.0734048 , 0.02676748, 0.02766727, 0.08999605, 0.03465028])

mse = tf.keras.losses.MeanSquaredError()

tf.config.experimental_run_functions_eagerly(True)

print(mse(targets, predictions))
print(mse(targets[:, tf.newaxis], predictions))
print(mse(targets, predictions[:, tf.newaxis]))
print(mse(targets[:, tf.newaxis], predictions[:, tf.newaxis]))

Output:

tf.Tensor(10867.5537109375, shape=(), dtype=float64)
tf.Tensor(10868.94140625, shape=(), dtype=float64)
tf.Tensor(10868.9404296875, shape=(), dtype=float64)
tf.Tensor(10867.552734375, shape=(), dtype=float64)

closed time in 7 days

w4nderlust

issue commenttensorflow/tensorflow

MSE loss computes different values depending on shape

@w4nderlust MSE loss expects inputs with the shape [batch_size, d0, .. dN]. The inputs must be atleast 2D. The function computes a mean along all except the batch axis, so the result you get will be one loss value per sample.

https://www.tensorflow.org/api_docs/python/tf/keras/losses/MSE?version=nightly

w4nderlust

comment created time in 7 days

PR opened tensorflow/tensorflow

Fix regression - keras fit class weights are being cast to int

PiperOrigin-RevId: 303009740 Change-Id: If06ef79162cc3569088feb0669a679954f0ec0e3

+2 -2

0 comment

2 changed files

pr created time in 7 days

create barnchtensorflow/tensorflow

branch : cherrypicks_TCQYU

created branch time in 7 days

fork pavithrasv/tensorflow

An Open Source Machine Learning Framework for Everyone

https://tensorflow.org

fork in 7 days

issue commenttensorflow/tensorflow

Keras connectivity metadata is not set correctly in TF 2.2.0rc0

@drasmuss We are going to be updating the code to remove metadata from all nested layers going forward. 'inbound_nodes' property on layers has been deprecated as it is an internal implementation detail.

drasmuss

comment created time in 7 days

pull request commenttensorflow/tensorflow

Added custom Huber loss

There is a Huber loss in keras losses already. If this is a different loss function may be it should be in the add ons repo?

Vishal-V

comment created time in 14 days

push eventtensorflow/cloud

Pavithra Vijay

commit sha b43fd206c6c22680fc32bcf22f4ab07c162f008f

Add setup files.

view details

Pavithra Vijay

commit sha 5ec88443638fb02f1c80e6d90b6582d11baec981

Updating version number and package classifiers.

view details

Pavithra Vijay

commit sha 90b4a900cdee01a23c3bffcd98f0835864cd9cf1

Merge pull request #4 from pavithrasv/master Add setup files.

view details

push time in 14 days

PR merged tensorflow/cloud

Add setup files.
+59 -0

0 comment

3 changed files

pavithrasv

pr closed time in 14 days

Pull request review commenttensorflow/cloud

Add setup files.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Setup script."""++from __future__ import absolute_import++from setuptools import find_packages+from setuptools import setup++VERSION = '1.0.0'+++setup(+    name='tensorflow-cloud',

Resolving this as per the offline discussion.

pavithrasv

comment created time in 14 days

Pull request review commenttensorflow/cloud

Add setup files.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Setup script."""++from __future__ import absolute_import++from setuptools import find_packages+from setuptools import setup++VERSION = '1.0.0'+++setup(+    name='tensorflow-cloud',+    version=VERSION,+    description='',+    url='https://github.com/tensorflow/cloud',+    author='The tensorflow cloud authors',+    author_email='tensorflow-cloud@google.com',+    license='Apache License 2.0',+    install_requires=[+        'docker',+        'google-api-python-client',+    ],+    extras_require={+        'tests': ['pytest',+                  'flake8',+                  'mock'],+    },+    classifiers=[+        'Programming Language :: Python',+        'Programming Language :: Python :: 3.6',+        'Operating System :: Unix',+        'Operating System :: Microsoft :: Windows',

No, removed this for now.

pavithrasv

comment created time in 15 days

Pull request review commenttensorflow/cloud

Add setup files.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Setup script."""++from __future__ import absolute_import++from setuptools import find_packages+from setuptools import setup++VERSION = '1.0.0'+++setup(+    name='tensorflow-cloud',+    version=VERSION,+    description='',+    url='https://github.com/tensorflow/cloud',+    author='The tensorflow cloud authors',+    author_email='tensorflow-cloud@google.com',+    license='Apache License 2.0',+    install_requires=[+        'docker',+        'google-api-python-client',+    ],+    extras_require={+        'tests': ['pytest',+                  'flake8',+                  'mock'],+    },+    classifiers=[+        'Programming Language :: Python',+        'Programming Language :: Python :: 3.6',

Removed this.

pavithrasv

comment created time in 15 days

Pull request review commenttensorflow/cloud

Add setup files.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Setup script."""++from __future__ import absolute_import++from setuptools import find_packages+from setuptools import setup++VERSION = '1.0.0'

Done.

pavithrasv

comment created time in 15 days

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 5ec88443638fb02f1c80e6d90b6582d11baec981

Updating version number and package classifiers.

view details

push time in 15 days

pull request commenttensorflow/tensorflow

Fixed missing export for Huber loss as tf.keras.losses.huber and tf.keras.metrics.huber

@mihaimaruseac can you advice on how API changes ie.pbtxt files can be updated?

srihari-humbarwadi

comment created time in 20 days

Pull request review commenttensorflow/tensorflow

Fixed missing export for Huber loss as tf.keras.losses.huber and tf.keras.metrics.huber

 def categorical_hinge(y_true, y_pred):   return math_ops.maximum(0., neg - pos + 1.)  +@keras_export('keras.metrics.huber', 'keras.losses.huber')

Can we only add it to losses namespace for now? Also you will need to update golden files '.pbtxt' files for API changes.

srihari-humbarwadi

comment created time in 20 days

issue commenttensorflow/tensorflow

How to deserialize from a dict with tf.keras.losses.get

Thank you the response @k-w-w and the quick PR @ashutosh1919 .

@k-w-w Yes, we can look into returning partial functions if class name is the name of a function. From the top of my head i think deserialize_keras_object may need to be modified for that.

jpatts

comment created time in 20 days

pull request commenttensorflow/tensorflow

Fixed missing export for Huber loss as tf.keras.losses.huber and tf.keras.metrics.huber

Why is this required? Can we use the Huber loss class instead? The other loss functions are exported because they were available historically.

srihari-humbarwadi

comment created time in 20 days

Pull request review commenttensorflow/tensorflow

Added example related to top_k parameter in tf.keras.metrics.Precision

 class Precision(Metric):   >>> m.result().numpy()   1.0 +  >>> # With top_k=2, it will calculate precision over y_true[:2] and y_pred[:2]+  >>> m = tf.keras.metrics.Precision(top_k=2)+  >>> _ = m.update_state([0, 0, 1, 1], [1, 1, 1, 1])+  >>> m.result().numpy()+  0.0

nit: Can you add a line break after this?

ashutosh1919

comment created time in 21 days

Pull request review commenttensorflow/tensorflow

Added example related to top_k parameter in tf.keras.metrics.Precision

 class Precision(Metric):   >>> m.result().numpy()   1.0 +  >>> # With top_k=2, it will calculate precision over y_true[:2] and y_pred[:2]+  >>> m = tf.keras.metrics.Precision(top_k=2)+  >>> _ = m.update_state([0, 0, 1, 1], [1, 1, 1, 1])+  >>> m.result().numpy()+  0.0+  >>> # With top_k=2, it will calculate precision over y_true[:4] and y_pred[:4]

top_k=4

ashutosh1919

comment created time in 21 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

 @keras_export('keras.applications.imagenet_utils.preprocess_input') def preprocess_input(x, data_format=None, mode='caffe'):   """Preprocesses a tensor or Numpy array encoding a batch of images."""+  if mode not in {'caffe', 'tf','torch'}:+    raise ValueError('Unknown mode ' + str(mode))

Yes.

jaketae

comment created time in 23 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

 @keras_export('keras.applications.imagenet_utils.preprocess_input') def preprocess_input(x, data_format=None, mode='caffe'):   """Preprocesses a tensor or Numpy array encoding a batch of images."""+  if mode not in {'caffe', 'tf','torch'}:+    raise ValueError('Unknown mode ' + str(mode))

Yes

jaketae

comment created time in 23 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

       {ret}    Raises:-      ValueError: In case of unknown `data_format` argument.+      ValueError: In case of unknown `mode` or `data_format` argument.

Please checkout how mobilenet reuses this docstring. mode is not an input in the preprocess_input function of mobilenet.

jaketae

comment created time in 23 days

PR opened tensorflow/cloud

Add setup files.
+55 -0

0 comment

2 changed files

pr created time in 23 days

push eventpavithrasv/cloud

Pavithra Vijay

commit sha b43fd206c6c22680fc32bcf22f4ab07c162f008f

Add setup files.

view details

push time in 23 days

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 80c4b4276da83894fbc72b341089ab60adc4977e

Adding setup files.

view details

push time in 23 days

push eventtensorflow/cloud

Pavithra Vijay

commit sha 8a9190f52524d73670be1edb1eaae14d8eb4c112

Adding support for streaming logs.

view details

Pavithra Vijay

commit sha d054d1c1473f51c1a18e9cf57bcad1db023026d9

Fixing exception.

view details

Pavithra Vijay

commit sha 0ddc272bde67f1d962713718a2cc530b02b3e8eb

Fix raises doc

view details

Pavithra Vijay

commit sha 0f0f366bb7adc76c6a86829ab85ba3ce73fe8d04

Update to using subprocess run.

view details

Pavithra Vijay

commit sha f8ff43486eaed801485f47d3b5dded7ab607151f

Merge pull request #3 from pavithrasv/master Adding support for streaming logs.

view details

push time in 23 days

PR merged tensorflow/cloud

Adding support for streaming logs.

Adding support for streaming job logs using subprocess and AI platform command.

+26 -5

1 comment

5 changed files

pavithrasv

pr closed time in 23 days

Pull request review commenttensorflow/cloud

Adding support for streaming logs.

 def _print_logs_info(job_id, project_id):           .format(job_id, project_id))  +def _stream_logs(job_id):+    """Streams job logs to stdout.++    Args:+        job_id: The job id to stream logs from.++    Raises:+        RuntimeError, if there are any errors from the streaming subprocess.+    """+    try:+        print('Streaming job logs: ')+        process = subprocess.Popen(

Updated.

pavithrasv

comment created time in 24 days

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 0f0f366bb7adc76c6a86829ab85ba3ce73fe8d04

Update to using subprocess run.

view details

push time in 24 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

 @keras_export('keras.applications.imagenet_utils.preprocess_input') def preprocess_input(x, data_format=None, mode='caffe'):   """Preprocesses a tensor or Numpy array encoding a batch of images."""+  if mode not in {'caffe', 'tf','torch'}:+    raise ValueError('Unknown mode ' + str(mode))

Please take a look at https://github.com/tensorflow/tensorflow/blob/aa070f5be6bfc55c25802540da149138696f965f/tensorflow/python/keras/applications/imagenet_utils_test.py#L29

We can add a simple test case for invalid mode here.

jaketae

comment created time in 24 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

 def validate_activation(classifier_activation, weights):     return    classifier_activation = activations.get(classifier_activation)-  if classifier_activation not in [+  if classifier_activation not in {

Thanks for the note! We don't need tests for this.

jaketae

comment created time in 24 days

issue commenttensorflow/tensorflow

AutoGraph error with FOR loop in Keras loss

Yes, i think it's a good idea to explore autographing custom Loss/Metric.

evgal

comment created time in 24 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

       {ret}    Raises:-      ValueError: In case of unknown `data_format` argument.+      ValueError: In case of unknown `mode` or `data_format` argument.

This cannot be added here as this doc is shared across all applications. This will have to be made a placeholder similar to {mode} / {ret} above.

jaketae

comment created time in 24 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

 @keras_export('keras.applications.imagenet_utils.preprocess_input') def preprocess_input(x, data_format=None, mode='caffe'):   """Preprocesses a tensor or Numpy array encoding a batch of images."""+  if mode not in {'caffe', 'tf','torch'}:+    raise ValueError('Unknown mode ' + str(mode))+
  • space
jaketae

comment created time in 24 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

 @keras_export('keras.applications.imagenet_utils.preprocess_input') def preprocess_input(x, data_format=None, mode='caffe'):   """Preprocesses a tensor or Numpy array encoding a batch of images."""+  if mode not in {'caffe', 'tf','torch'}:+    raise ValueError('Unknown mode ' + str(mode))

Can you add a test for this?

jaketae

comment created time in 24 days

Pull request review commenttensorflow/tensorflow

Minor refactoring and robust mode check for imagenet_utils.py

 def validate_activation(classifier_activation, weights):     return    classifier_activation = activations.get(classifier_activation)-  if classifier_activation not in [+  if classifier_activation not in {

Why do you want to change this to a dict?

jaketae

comment created time in 24 days

pull request commenttensorflow/cloud

Adding support for streaming logs.

Thank you Francois!

pavithrasv

comment created time in a month

Pull request review commenttensorflow/cloud

Adding support for streaming logs.

 def _print_logs_info(job_id, project_id):           .format(job_id, project_id))  +def _stream_logs(job_id):+    """Streams job logs to stdout.++    Args:+        job_id: The job id to stream logs from.++    Raises:+        RuntimeError, if there are any errors from the streaming subprocess.+    """+    try:+        print('Streaming job logs: ')+        process = subprocess.Popen(

run is available only in Python >= 3.5 hence i used Popen. Afaik the code so far works in python >= 2.7. I am not sure if we want to add version restriction for this since run is only a simplified version of Popen.

pavithrasv

comment created time in a month

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 0ddc272bde67f1d962713718a2cc530b02b3e8eb

Fix raises doc

view details

push time in a month

push eventpavithrasv/cloud

Pavithra Vijay

commit sha d054d1c1473f51c1a18e9cf57bcad1db023026d9

Fixing exception.

view details

push time in a month

PR opened tensorflow/cloud

Adding support for streaming logs.
+32 -2

0 comment

2 changed files

pr created time in a month

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 8a9190f52524d73670be1edb1eaae14d8eb4c112

Adding support for streaming logs.

view details

push time in a month

pull request commenttensorflow/cloud

Add save, load example and update examples with different distributio…

Thank you Francois!

pavithrasv

comment created time in a month

push eventtensorflow/cloud

Pavithra Vijay

commit sha 6a00fc61779d92c00a8ca766b337a9c42384289d

Add save, load example and update examples with different distribution strategies.

view details

Pavithra Vijay

commit sha f0572d77ebb3ffc5f46ac0d77aea790dd90f8510

Make the local env TF installation optional.

view details

Pavithra Vijay

commit sha ba5447e7c2d77706aeb59ed30322ecaaf11089c8

Adding a folder for intergration tests.

view details

Pavithra Vijay

commit sha 890afaca57a5d5406328e1eb3fccd251f1c6727a

Merge pull request #2 from pavithrasv/master Add save, load example and update examples with different distributio…

view details

push time in a month

PR merged tensorflow/cloud

Add save, load example and update examples with different distributio…

…n strategies.

+210 -37

1 comment

11 changed files

pavithrasv

pr closed time in a month

Pull request review commenttensorflow/cloud

Add save, load example and update examples with different distributio…

+# Copyright 2020 Google LLC. All Rights Reserved.

Sounds good, added a folder.

pavithrasv

comment created time in a month

push eventpavithrasv/cloud

Pavithra Vijay

commit sha ba5447e7c2d77706aeb59ed30322ecaaf11089c8

Adding a folder for intergration tests.

view details

push time in a month

pull request commenttensorflow/cloud

Add save, load example and update examples with different distributio…

Made local TF installation optional in the latest commit. PTAL.

pavithrasv

comment created time in a month

push eventpavithrasv/cloud

Pavithra Vijay

commit sha f0572d77ebb3ffc5f46ac0d77aea790dd90f8510

Make the local env TF installation optional.

view details

push time in a month

Pull request review commenttensorflow/tensorflow

Categorical Hinge Loss Doc Updated

 class CategoricalHinge(LossFunctionWrapper):   """Computes the categorical hinge loss between `y_true` and `y_pred`.    `loss = maximum(neg - pos + 1, 0)`-  where `neg = sum(y_true * y_pred)` and `pos = maximum(1 - y_true)`+  where `neg=maximum((1-y_true)*y_pred) and pos=sum(y_true*y_pred)`

Thank you, can you make the same change in the categorical_hinge function below?

ayushmankumar7

comment created time in a month

Pull request review commenttensorflow/cloud

Add save, load example and update examples with different distributio…

 def get_wrapped_entry_point(entry_point,         'import tensorflow as tf\n',         strategy,         'tf.distribute.experimental_set_strategy(strategy)\n',-        # Add user's code+        # Add user's code.+        # We are using exec here to execute the user code object.+        # This will support use case where the user's program has a+        # main method.

I left the exec call here as copying the user code and executing exec would pretty much do the same thing. exec will only be just as unsafe as copying and executing.

pavithrasv

comment created time in a month

Pull request review commenttensorflow/cloud

Add save, load example and update examples with different distributio…

 from tensorflow_cloud import machine_config from tensorflow_cloud import run +# # Automated CPU strategy: CPU chief config and no workers+# run.run(

Yes, this is until we figure out how to run integration tests. Did not want to create a separate file for every workflow combination.

pavithrasv

comment created time in a month

Pull request review commenttensorflow/cloud

Add save, load example and update examples with different distributio…

+# Copyright 2020 Google LLC. All Rights Reserved.

This should be an integration test. For now i am adding them as scripts here as we do not yet know integration test framework for this repo.

pavithrasv

comment created time in a month

PR opened tensorflow/cloud

Add save, load example and update examples with different distributio…

…n strategies.

+191 -29

0 comment

9 changed files

pr created time in a month

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 6a00fc61779d92c00a8ca766b337a9c42384289d

Add save, load example and update examples with different distribution strategies.

view details

push time in a month

issue commenttensorflow/tensorflow

wrong doc for categorical_hinge loss

@ayushmankumar7 feel free to send me a PR.

openyuanmu

comment created time in a month

Pull request review commenttensorflow/tensorflow

Edited docstring, y_pred for SparseCategoricalAcc

 def __init__(self, name='categorical_accuracy', dtype=None): @keras_export('keras.metrics.SparseCategoricalAccuracy') class SparseCategoricalAccuracy(MeanMetricWrapper):   """Calculates how often predictions matches integer labels.--  You can provide logits of classes as `y_pred`, since argmax of+  +  For example, if `y_true` is `[[2], [1]]` and `y_pred` is

You can use np.equal in place of that. Also weighting denom should be sum of sample_weights, not n_pred. Thank you!

jaketae

comment created time in a month

Pull request review commenttensorflow/tensorflow

Edited docstring, y_pred for SparseCategoricalAcc

 def __init__(self, name='categorical_accuracy', dtype=None): @keras_export('keras.metrics.SparseCategoricalAccuracy') class SparseCategoricalAccuracy(MeanMetricWrapper):   """Calculates how often predictions matches integer labels.--  You can provide logits of classes as `y_pred`, since argmax of+  +  For example, if `y_true` is `[[2], [1]]` and `y_pred` is

The sentence here is just putting the below example into words. We are not really explaining how we got the values here ie the formula used etc. This explanation is something I have been meaning to add to metrics soon. We have added such documentation to losses.py already. Please take a look.

jaketae

comment created time in a month

Pull request review commenttensorflow/tensorflow

Edited docstring, y_pred for SparseCategoricalAcc

 def __init__(self, name='categorical_accuracy', dtype=None): @keras_export('keras.metrics.SparseCategoricalAccuracy') class SparseCategoricalAccuracy(MeanMetricWrapper):   """Calculates how often predictions matches integer labels.--  You can provide logits of classes as `y_pred`, since argmax of+  +  For example, if `y_true` is `[[2], [1]]` and `y_pred` is

This change can be omitted since the example below conveys the same information.

jaketae

comment created time in a month

issue closedtensorflow/tensorflow

Return predictions with .evaluate()

I work with several custom metrics I compute from model's inference-mode predictions; problem is, this requires me to hand-code loss computation to avoid running .evaluate() along .predict() (doubling validation time), which may include weight and activity regularizers.

It'd be quite helpful to have an option for model.evaluate() to return predictions along computed metrics (e.g. evaluate(return_predictions=True)).

NOTE: this is a feature request, concerning the evaluate method of keras.Model or tf.keras.Model.

closed time in a month

OverLordGoldDragon

issue commenttensorflow/tensorflow

Return predictions with .evaluate()

Thank you for the request @OverLordGoldDragon. We do not plan to add this to the built in loops. You can implement this in a custom evaluate loop by keeping track of the batch prediction values.

OverLordGoldDragon

comment created time in a month

issue closedtensorflow/tensorflow

Feature request: RecallAtPrecision Metric

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Mac OS
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary): pip install
  • TensorFlow version (use command below): 2.1.0
  • Python version: (3, 6, 8, 'final', 0)
  • Bazel version (if compiling from source): NA
  • GCC/Compiler version (if compiling from source): NA
  • CUDA/cuDNN version: NA
  • GPU model and memory: NA
  • Exact command to reproduce: NA

Feature request: tf.keras.metrics.RecallAtPrecision

Thank you all for including PrecisionAtRecall, would it be possible to also add in the inverse, Recall At Precision? I tried adding a custom metric, but its rather difficult to replicate the pattern, since I cannot inherit the SensitivitySpecificityBase class defined in the source code.

Thank you!

closed time in a month

tangbyron

issue commenttensorflow/tensorflow

Feature request: RecallAtPrecision Metric

RecallAtPrecision has been added.

tangbyron

comment created time in a month

issue commenttensorflow/tensorflow

Add a Model.fit-like API for seq2seq models

Adding @qlzh727 who is a seq2seq model expert to advice on this. Thank you!

luozhouyang

comment created time in a month

issue commenttensorflow/tensorflow

Keras with multiple outputs: cannot evaluate a metric without associated loss

Thank you @durandg12. This is by design today but we can look into whether the behavior can be changed. Marked this as a feature request.

durandg12

comment created time in a month

PR closed tensorflow/tensorflow

add documentation for Discrepancy between keras.layers.Reshape and tf.keras.layers.Reshape awaiting review cla: yes comp:keras prtype:bugfix size:XS

resolves #36428

+9 -0

3 comments

1 changed file

rushabh-v

pr closed time in a month

push eventtensorflow/cloud

Pavithra Vijay

commit sha 90b6bdca3de7df385884bbaa1aaae5fe0fcb8972

Initial commit.

view details

Pavithra Vijay

commit sha c4e769e004b4b41860186f355adf6337f8ab0d45

Add doc strings for all methods, optionally create wrapped_entry_point_script, update unique id generation.

view details

Pavithra Vijay

commit sha 63ba5971e78062e1ff0c8401e612bdd62f89fca4

Update deply_job docs.

view details

Pavithra Vijay

commit sha 15305cbf044ec6994a442094230b5581a0fb4765

Remove region from the API as it is GCP specific.

view details

Pavithra Vijay

commit sha c72f8c4b6baf925aea976b05de9574cb08e65971

Merge pull request #1 from pavithrasv/master Initial commit.

view details

push time in a month

PR merged tensorflow/cloud

Initial commit.

Adds run API. Adds support for workflow where a python program can be passed as an entry_point to run.

+2680 -1

2 comments

24 changed files

pavithrasv

pr closed time in a month

pull request commenttensorflow/cloud

Initial commit.

Thank you Francois!

pavithrasv

comment created time in a month

issue commenttensorflow/tensorflow

tf.keras.backend.set_floatx() causing ValueError (dtype conversion error) while computing tf.keras.metrics.*

Thank you @MarkDaoust. It should be cast to the predictions' dtype. If anyone would like to work on the fix please feel free to send me a PR.

Hemal-Mamtora

comment created time in a month

push eventpavithrasv/cloud

Pavithra Vijay

commit sha 15305cbf044ec6994a442094230b5581a0fb4765

Remove region from the API as it is GCP specific.

view details

push time in 2 months

issue commentkeras-team/keras

invalid literal for int() with base 10: 'Participant1'

Sorry about the delay, can you share the code that you ran to get the above error, if possible as standalone code snippet that would help us repro the issue?

sekti92

comment created time in 2 months

issue closedtensorflow/tensorflow

Why K.mean is used in tf.keras.losses.binary_crossentropy ?

In line 993 of the code of tf.keras.losses.binary_crossentropy, K.mean is called on axis -1 of K.binary_crossentropy(y_true, y_pred, from_logits=from_logits).

I wonder why there is this K.mean call and why tf.keras.losses.binary_crossentropy doesn't simply return K.binary_crossentropy(y_true, y_pred, from_logits=from_logits).

On the contrary, tf.keras.losses.categorical_crossentropy and tf.keras.losses.sparse_categorical_crossentropy just return the call to their tf.keras.backend equivalent.

I think it may be inconsistent and misleading, especially because tf.keras.losses.categorical_crossentropy and tf.keras.backend.categorical_crossentropy behave similarly, but not tf.keras.losses.binary_crossentropy and tf.keras.backend.binary_crossentropy, so higher level objects like tf.keras.metrics.CategoricalCrossentropy may not work as one would expect.

closed time in 2 months

durandg12

pull request commenttensorflow/cloud

Initial commit.

Thank you for the quick review Francois, updated the docs and addressed the comments. PTAL!

pavithrasv

comment created time in 2 months

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):+    if accl_type == 'CPU':+        return 'ACCELERATOR_TYPE_UNSPECIFIED'+    if accl_type == 'K80':+        return 'NVIDIA_TESLA_K80'+    if accl_type == 'P100':+        return 'NVIDIA_TESLA_P100'+    if accl_type == 'V100':+        return 'NVIDIA_TESLA_V100'+    if accl_type == 'P4':+        return 'NVIDIA_TESLA_P4'+    if accl_type == 'T4':+        return 'NVIDIA_TESLA_T4'+    else:+        raise ValueError('Invalid accelerator type.')+++def get_machine_type(cpu_cores, memory):+    config = (cpu_cores, memory)+    if config == (4, 15):+        return 'n1-standard-4'+    if config == (8, 30):+        return 'n1-standard-8'+    if config == (16, 60):+        return 'n1-standard-16'+    if config == (32, 120):+        return 'n1-standard-32'+    if config == (64, 240):+        return 'n1-standard-64'+    if config == (96, 360):+        return 'n1-standard-96'+    if config == (2, 13):+        return 'n1-highmem-2'+    if config == (4, 26):+        return 'n1-highmem-4'+    if config == (8, 52):+        return 'n1-highmem-8'+    if config == (16, 104):+        return 'n1-highmem-16'+    if config == (32, 208):+        return 'n1-highmem-32'+    if config == (64, 416):+        return 'n1-highmem-64'+    if config == (96, 624):+        return 'n1-highmem-96'+    if config == (16, 14.4):+        return 'n1-highcpu-16'+    if config == (32, 28.8):+        return 'n1-highcpu-32'+    if config == (64, 57.6):+        return 'n1-highcpu-64'+    if config == (96, 86.4):+        return 'n1-highcpu-96'+    else:+        raise ValueError('Invalid machine type.')

This actually should never happen, this is not a public API. 'Validate_machine_configurations' API above validates machine type at creation time and raises error if the input is invalid. That also provides link to the valid machine types list.

pavithrasv

comment created time in 2 months

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):+    if accl_type == 'CPU':+        return 'ACCELERATOR_TYPE_UNSPECIFIED'+    if accl_type == 'K80':+        return 'NVIDIA_TESLA_K80'+    if accl_type == 'P100':+        return 'NVIDIA_TESLA_P100'+    if accl_type == 'V100':+        return 'NVIDIA_TESLA_V100'+    if accl_type == 'P4':+        return 'NVIDIA_TESLA_P4'+    if accl_type == 'T4':+        return 'NVIDIA_TESLA_T4'+    else:+        raise ValueError('Invalid accelerator type.')

This actually should never happen, this is not a public API. Accelerator type class validates accelerator type at creation time and raises error if the input is invalid.

pavithrasv

comment created time in 2 months

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):+    if accl_type == 'CPU':+        return 'ACCELERATOR_TYPE_UNSPECIFIED'+    if accl_type == 'K80':+        return 'NVIDIA_TESLA_K80'+    if accl_type == 'P100':+        return 'NVIDIA_TESLA_P100'+    if accl_type == 'V100':+        return 'NVIDIA_TESLA_V100'+    if accl_type == 'P4':+        return 'NVIDIA_TESLA_P4'+    if accl_type == 'T4':+        return 'NVIDIA_TESLA_T4'+    else:+        raise ValueError('Invalid accelerator type.')+++def get_machine_type(cpu_cores, memory):

Done.

pavithrasv

comment created time in 2 months

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import google.auth+++def get_project_name():+    # https://google-auth.readthedocs.io/en/latest/reference/google.auth.html+    _, project_id = google.auth.default()+    if project_id is None:+        raise Exception('Could not determine the GCP project id.')++    return project_id+++def validate_machine_configuration(+        cpu_cores, memory, accelerator_type, accelerator_count):+    valid_configurations = _get_valid_machine_configurations()+    current_config = (+        cpu_cores, memory, accelerator_type.value, accelerator_count)+    if current_config not in valid_configurations:+        raise ValueError(+            'Invalid machine configuration: cpu_cores:{}, memory:{}, '+            'accelerator_type:{}, accelerator_count:{}. Please see the '+            'following AI platform comptibility table for all valid '+            'configurations: '+            'https://cloud.google.com/ml-engine/docs/using-gpus#'+            'compute-engine-machine-types-with-gpu'.format(+                cpu_cores, memory, str(accelerator_type), accelerator_count))+++def get_region():+    return 'us-central1'+++def get_accelerator_type(accl_type):

Done.

pavithrasv

comment created time in 2 months

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import os+import sys++from . import containerize+from . import deploy+from . import gcp+from . import machine_config+from . import package+from . import preprocess+from . import validate+++# Flag which indicates whether current process is running in a cloud+# environment created by the `cloud.run` API.+_IS_RUNNING_REMOTELY = False+++def _is_running_remotely():+    return _IS_RUNNING_REMOTELY+++def _set_running_remotely(value):+    global _IS_RUNNING_REMOTELY+    _IS_RUNNING_REMOTELY = value+++def run(+    entry_point,+    requirements_txt=None,+    distribution_strategy='auto',+    docker_base_image=None,+    chief_config='auto',+    worker_config='auto',+    worker_count=0,+    region=None,+    entry_point_args=None,+    stream_logs=False,+):+    """Runs your Tensorflow code in Google Cloud Platform.++    # Arguments:+        entry_point: String. Python file path to the file that contains the+            TensorFlow code.+            Note: This path must be in the current working directory tree.+            Example: 'train.py', 'training/mnist.py'+        requirements_txt: Optional string. File path to requirements.txt file+            containing aditionally pip dependencies if any.+            Note: This path must be in the current working directory tree.+            Example: 'requirements.txt', 'deps/reqs.txt'+        distribution_strategy: 'auto' or None. Defaults to 'auto'.+            'auto' means we will take care of creating a Tensorflow+            distribution strategy instance based on the machine configurations+            you have provided using the `chief_config`, `worker_config` and+            `worker_count` params.+            - If the number of workers > 0, we will use+                `tf.distribute.experimental.MultiWorkerMirroredStrategy`.+            - If number of GPUs > 0, we will use+                `tf.distribute.MirroredStrategy`+            If you have created a distribution strategy instance in your script+            already, please set `distribution_stratgey` as None here.+            For example, if you are using `tf.keras` custom training loops,+            you will need to create a strategy in the script for distributing+            the dataset.+        docker_base_image: Optional base docker image to use. Defaults to None.+            Example: 'gcr.io/my_gcp_project/deep_learning:v2'+            If a base docker image is not provided here, we will use a+            Tensorflow docker image (https://www.tensorflow.org/install/docker)+            as the base image. The version of TensorFlow and Python in that+            case will match your local environment.+        chief_config: Optional `MachineConfig` that represents the+            configuration for the chief worker in a distribution cluster.+            Defaults to 'auto'. 'auto' maps to a standard gpu config such as+            `COMMON_MACHINE_CONFIGS.P100_1X` (8 cpu cores, 30GB memory,+            1 Nvidia Tesla P100).+        worker_config: Optional `MachineConfig` that represents the+            configuration for the general workers in a distribution cluster.+            Defaults to 'auto'. 'auto' maps to a standard gpu config such as+            `COMMON_MACHINE_CONFIGS.P100_1X` (8 cpu cores, 30GB memory,+            1 Nvidia Tesla P100).+        worker_count: Optional integer that represents the number of general+            workers in a distribution cluster. Defaults to 0. This count does+            not include the chief worker.+        region: Optional string. Cloud region in which to submit the+            job. Defaults to 'us-central1' for GCP.+        entry_point_args: Optional list of strings. Defaults to None.+            Command line arguments to pass to the `entry_point` program.+        stream_logs: Boolean flag which when enabled streams logs back from+            the cloud job.+    """+    # If code is triggered in a cloud environment, do nothing.+    if _is_running_remotely():+        return+    _set_running_remotely(True)++    # Get defaults.+    if chief_config == 'auto':+        chief_config = machine_config.COMMON_MACHINE_CONFIGS['P100_1X']+    if worker_config == 'auto':+        worker_config = machine_config.COMMON_MACHINE_CONFIGS['P100_1X']+    region = region or gcp.get_region()+    dst_path_prefix = '/app/'+    docker_registry = 'gcr.io/{}'.format(gcp.get_project_name())++    # Run validations.+    validate.validate(+        entry_point, distribution_strategy, requirements_txt,+        chief_config, worker_config, worker_count, region,+        entry_point_args, stream_logs)++    # Create the script to run (starter_script).+    # Make the `entry_point` cloud and distribution ready.+    startup_script = preprocess.get_startup_script(+        entry_point, chief_config, worker_count, distribution_strategy)++    # Get all the files, that we need to package, mapped to the dst location.+    # This will include the startup script, requirements_txt, dockerfile,+    # files in the entry_point dir.+    dockerfile, file_map = containerize.get_file_map(+        entry_point, startup_script, chief_config, requirements_txt,+        dst_path_prefix, docker_base_image)++    # Create a tarball with the files.+    tarball = package.get_tarball(file_map)++    # Create docker image.+    docker_img = containerize.get_docker_image(docker_registry, tarball)++    # Delete all the temporary files we created.

Made 'wrapped_entry_point' script creation optional - one less deletion to worry about. I am writing some examples and detailed usages doc - will mention these in that.

pavithrasv

comment created time in 2 months

Pull request review commenttensorflow/cloud

Initial commit.

+# Copyright 2020 Google LLC. All Rights Reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from __future__ import absolute_import+from __future__ import division+from __future__ import print_function++import os+import sys++from . import containerize+from . import deploy+from . import gcp+from . import machine_config+from . import package+from . import preprocess+from . import validate+++# Flag which indicates whether current process is running in a cloud+# environment created by the `cloud.run` API.+_IS_RUNNING_REMOTELY = False+++def _is_running_remotely():+    return _IS_RUNNING_REMOTELY+++def _set_running_remotely(value):+    global _IS_RUNNING_REMOTELY+    _IS_RUNNING_REMOTELY = value+++def run(+    entry_point,+    requirements_txt=None,+    distribution_strategy='auto',+    docker_base_image=None,+    chief_config='auto',+    worker_config='auto',+    worker_count=0,+    region=None,+    entry_point_args=None,+    stream_logs=False,+):+    """Runs your Tensorflow code in Google Cloud Platform.++    # Arguments:

Fixed, using Args: throughtout.

pavithrasv

comment created time in 2 months

more