profile
viewpoint
Dan Moldovan mdanatg @google USA Software Engineer

tensorflow/examples 3283

TensorFlow examples

google/tangent 2052

Source-to-Source Debuggable Derivatives in Pure Python

mdanatg/community 0

Stores documents used by the TensorFlow developer community

mdanatg/copybara 0

Copybara: A tool for transforming and moving code between repositories.

issue commenttensorflow/tensorflow

`nan` gradient when `tf.where` is used

@tushar-dalal The challenge is that verifying for such NaN inputs can be taking on performance. When debugging, tf.debugging.check_numerics can indeed help with that.

@unicorn-io Feel free to tackle it, but note that it's extremely challenging to solve. That said, there was a PR (#38467) to add a warning message to the docs of tf.where, it would be useful to revive it.

0x0badc0de

comment created time in 2 days

issue commenttensorflow/tensorflow

OperatorNotAllowedInGraphError error in Tensorflow 2.0

Sorry for the delay. @omalleyt12, any thoughts on that error?

payne4handsome

comment created time in 3 days

issue commenttensorflow/tensorflow

Wrong result when calling 'multi-line' lambda inside @tf.function

This looks like an underlying Python bug:

l = (
     lambda y: lambda x: x*y
     - y
    )

import inspect
print(inspect.getsource(l))

Output:

     lambda y: lambda x: x*y

It looks like the parser misses the second line.

To fix this, we'll need to work around this bug.

andrewsali

comment created time in 5 days

issue commenttensorflow/tensorflow

autograph for bool logic control incorrect

+1 to @yongtang 's note

I recommend using the value directly: if is_zero:. You can also use an equality check like if is_zero == True:, but there's no obvious advantage to it.

Indeed, autograph doesn't implement is and strictly speaking I'm not sure that it can, judging from the SO answer, because the semantics of a is True mean "a is exactly the singleton object True", in other words id(a) == id(True). Take for example another True-like object:

np_true = np.array(True)

bool(np_true)
# True

np_true == True
# True

np_true is True
# False
w19787

comment created time in 5 days

issue commenttensorflow/tensorflow

InaccessibleTensorError in graph mode with for loop itteration (different sized tensors) ( tf.image.resize)

Agreed, perhaps raising an error would be less confusing instead. The choice for int32 is because int32 and float32 are the default data types in TF (used e.g. when creating tf.constant).

bela127

comment created time in 6 days

issue commenttensorflow/tensorflow

InaccessibleTensorError in graph mode with for loop itteration (different sized tensors) ( tf.image.resize)

I see. Although sometimes you can downcast a Tensor to a Python int (see tf.get_static_value), that only works for simple values like constants, and won't work for outputs of more complicated ops. In our case, int(tensor) is just a type cast, that is tf.cast(tensor, tf.int32) - perhaps that was the source of the confusion, and the reason why I suggested avoiding to wrap self.count into a tf.constant in the first place.

bela127

comment created time in 8 days

issue commenttensorflow/tensorflow

InaccessibleTensorError in graph mode with for loop itteration (different sized tensors) ( tf.image.resize)

Yes, list manipulation with uneven tensors is not well supported at the moment. We're looking into fixing this. For now, the easiest way to make Scalar1 work, which I think is the most idiomatic version, is to avoid turning self.count into a Tensor.

Note that if self.count ever becomes a truly dynamic tensor, then you won't be able to use a Python list for self.sized_images - Python lists can't change length in TensorFlow.

An alternative to using lists or TensorArray is to use tf.RaggedTensor, which can hold elements of uneven size, and will work with a dynamic self.count as well.

bela127

comment created time in 8 days

issue commenttensorflow/tensorflow

WARNING:tensorflow:AutoGraph could not transform and will run it as-is. Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.

@KaticaR thank you for the logs, they seem to indicate an incompatibility with the toolchain, though it's unclear which piece. At any rate, the faulting piece was refactored recently. If you have the chance, please retry with tf-nightly.

nsssss

comment created time in 10 days

pull request commentdionhaefner/pyhpc-benchmarks

Use XLA compilation with tf.function

cc @cheshire Yep the GPU numbers are interesting...

dionhaefner

comment created time in 11 days

pull request commentdionhaefner/pyhpc-benchmarks

Use XLA compilation with tf.function

Yes, it forces XLA and raises an error if it could not be compiled. Indeed, the performance should be similar to JAX.

mdanatg

comment created time in 11 days

PR opened dionhaefner/pyhpc-benchmarks

Use XLA compilation with tf.function

The default executor is significantly underpowered for scalar computations.

+1 -1

0 comment

1 changed file

pr created time in 12 days

push eventmdanatg/pyhpc-benchmarks

Dan Moldovan

commit sha 705f76734a6c09b9268fa5c0be4e610292fe8362

Update eos_tensorflow.py Use XLA compilation with tf.function. The default executor is significantly underpowered for scalar computations.

view details

push time in 12 days

fork mdanatg/pyhpc-benchmarks

A suite of benchmarks to test the sequential CPU and GPU performance of most popular high-performance libraries for Python.

fork in 12 days

push eventmdanatg/Astronet-Triage

Dan Moldovan

commit sha 0d0df5aedfe32cc0d180a44f035f7fecb7fabfc8

Port more examples from Liang dataset.

view details

push time in 15 days

push eventmdanatg/Astronet-Triage

Dan Moldovan

commit sha 64227b9160219948a6a8ccbedbecc6ec3724d016

Include PC from Liang dataset.

view details

Dan Moldovan

commit sha 69f3350e16af196f5cfe5d4d5b1f4a6fb6611fa6

Fix file paths.

view details

Dan Moldovan

commit sha 00e0994e42a9d04e94ece899f9a18b63a2a3f703

Fix file paths.

view details

Dan Moldovan

commit sha 4026136d63366f43ca1b2709ecda63418548dce8

Fixes to the merging of old TCE data.

view details

Dan Moldovan

commit sha 4525694c63f8fce30d1413dd411dfa283589e13c

Port more examples from Liang dataset.

view details

push time in 15 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

Yep - looks like the tests passed! Consider updating the PR title.

bhack

comment created time in 16 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

I'm not sure, actually. Usually it picks up immediately.

bhack

comment created time in 16 days

Pull request review commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

 def compute(i, tas):       elems_value_flat = _elems_value_batchable_to_flat(elems_value_batchable,                                                         elems_flat_signature)       elems_value = elems_unflatten(elems_value_flat)+      autographed_fn = autograph.tf_convert(fn, autograph_ctx.control_status_ctx())+      result_value = autographed_fn(elems_value)       result_value = fn(elems_value)

Remove duplicate line - we must not call the original fn again.

bhack

comment created time in 16 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

I can trigger a build which should finish faster.

bhack

comment created time in 16 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

I don't think that we could just use result_value = autograph.tf_convert(fn(elems_value), autograph_ctx.control_status_ctx())

Cause we get: TypeError: Could not build a TypeSpec for <function convert.<locals>.decorator.<locals>.wrapper at 0x7f651f655e18> with type function

Correct, tf_convert doesn't call the function, it just translates it.

So we need something like:

autographed_fn = autograph.tf_convert(fn, autograph_ctx.control_status_ctx())
result_value = autographed_fn(elems_value)
bhack

comment created time in 16 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

Not sure. It's possible that map_fn switches to eager mode even when you have run_in_graph_and_eager_modes. Using tf.function will make sure that's not the case.

bhack

comment created time in 16 days

Pull request review commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

 def testMap_MultiInputSameStructureOutput(self):     self.assertAllEqual(-nums, received[1])     self.assertAllEqual(nums, received[2]) +  #@test_util.run_in_graph_and_eager_modes+  @test_util.run_deprecated_v1+  def testMap_autograph_indirect(self):+    def test_function(x):+      cond = constant_op.constant(-1)+      if cond == 0:+        result = x+      else:+        result = x+      return result++    @api.convert(recursive=False) 

Use def_function.function here (imported as from tensorflow.python.eager import def_function)

bhack

comment created time in 16 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

This is the most practical way I think: https://www.tensorflow.org/install/source#docker_linux_builds.

Once it's working you should be able to run something like bazel test //tensorflow/python/kernel_tests:map_fn_test

The initial call to bazel can take a while since it needs to rebuild everything. Afterwards it should be much faster.

bhack

comment created time in 17 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

I think it's safest to do that. It might work with a direct import as well, but there are many circular dependencies there.

bhack

comment created time in 17 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

Sure. So map_fn needs to apply autograph to the target function before calling it, here:

https://github.com/tensorflow/tensorflow/blob/8e8c67c3375da3fe8b44e7c11eb1d3fbb2eaa41c/tensorflow/python/ops/map_fn.py#L480

The right way to do that is shown in this code that does the same for Dataset.map:

https://github.com/tensorflow/tensorflow/blob/8e8c67c3375da3fe8b44e7c11eb1d3fbb2eaa41c/tensorflow/python/data/ops/dataset_ops.py#L3223

bhack

comment created time in 17 days

issue commenttensorflow/tensorflow

Tensorflow 2.0 Preview - tf.function error when wrapping a function that works

Looks like a bug in map_fn, which doesn't apply autograph to its function. The partial part was a bit of a red herring.

botev

comment created time in 17 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call tf.map_fn

It's probably not that frequent, since the functions passed to map_fn are usually simple and don't use control flow. I think it's useful to fix though.

bhack

comment created time in 17 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call partial

I don't think that's caused by partial. The problem is that map_fn doesn't call autograph (see below). That's definitely a bug - feel free to file an issue, and if you're interested to help fixing it, I can give a few pointers as well.

A workaround is to wrap the target of map_fn in tf.function:

@tf.function  # Fails without this, but shouldn't.
def test_function(x):
    cond = tf.constant(-1)

    if cond == 0:
        result = x
    else:
        result = x

    return result

@tf.function
def partial_call(x):
    tf.map_fn(test_function, x)

X = tf.random.uniform(shape=(10, 224, 224, 3))
partial_call(X)
bhack

comment created time in 17 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call partial

I see. Do you have a small reproducer in a colab, using @tf.function as decorator? Something simple like this test would do. It might be caused by keyword arguments, if I read through the comments correctly.

bhack

comment created time in 17 days

pull request commenttensorflow/tensorflow

Add a decorated function test that call partial

I think the issue was fixed a while ago, but the issue ID was never tagged by the fix. I definitely expect the test to pass. Have you seen recent cases when it doesn't work?

bhack

comment created time in 17 days

Pull request review commenttensorflow/tensorflow

Add a decorated function test that call partial

 def test_fn(x, y, z):         options=DEFAULT_RECURSIVE)     self.assertEqual((1, 2, 3), self.evaluate(x)) +  @test_util.run_v1_only('b/120545219')+  def test_converted_call_functools_partial_decorated(self):++    api.converted_call(options=DEFAULT_RECURSIVE)

Got it, noprob. I think we still need the @ to make it a decorator: @api.convert(...)?

bhack

comment created time in 17 days

Pull request review commenttensorflow/tensorflow

Add a decorated function test that call partial

 def test_fn(x, y, z):         options=DEFAULT_RECURSIVE)     self.assertEqual((1, 2, 3), self.evaluate(x)) +  @test_util.run_v1_only('b/120545219')+  def test_converted_call_functools_partial_decorated(self):++    api.converted_call(options=DEFAULT_RECURSIVE)

This statement has no effect - perhaps you meant it to be a decorator? At any rate, converted_call is not intended to be a decorator, but you can still use it as x = api.converted_call(test_call_partial, ...)

bhack

comment created time in 17 days

issue commenttensorflow/tensorflow

TypeError: len is not well defined for symbolic Tensors. (transpose:0) Please call `x.shape` rather than `len(x)` for shape information.

keras.preprocessing.pad_sequences expects a list of lists, and doesn't work with tensor inputs, which is what gets produced by tf.transpose. Using tf.pad is the right workaround in this case.

menon92

comment created time in 18 days

issue commenttensorflow/tensorflow

InaccessibleTensorError: The tensor 'Tensor("Tile:0", shape=(None, 3), dtype=float32)' cannot be accessed here: it is defined in another function or code block.

I'm hoping that in a future version we'll support the plain list version, because right now things are a bit clunky to use.

For the first solution, the trouble in this case seems to be that TensorArray only supports elements of the same shape. But I think we can make it work by tiling inside the array and avoiding the tf.tile. I'm not sure this does exactly what you need, but here's a version that produces results:

    new_batch = tf.TensorArray(tf.float32, size=batch_size)
    x = tf.expand_dims(x, axis=-2)
    for i in tf.range(batch_size):
        tensor_list = tf.TensorArray(tf.float32, size=0, dynamic_size=True)
        for j in tf.range(seq_len):
            # tiled = tf.tile(x[i][j], [dims[i][j][0], 1])
            for k in tf.range(dims[i][j][0]):
              tensor_list = tensor_list.write(tensor_list.size(), x[i][j])
        new_tensor = tensor_list.stack()
        new_tensor = tf.pad(new_tensor, [[0,pad_sizes[i][0]], [0,0], [0, 0]])
        new_batch = new_batch.write(i, new_tensor)
    return new_batch.stack()

The second solution only works in TF 2.2 (that's when set_loop_options was added). I fudged the example by hardcoding some shapes, you'd need to use the proper ones in those tf.zeros and shape_invariants:

    new_batch = tf.zeros([0, 6, 3])
    x = tf.expand_dims(x, axis=-2)
    for i in tf.range(batch_size):
        tf.autograph.experimental.set_loop_options(
            shape_invariants=[(new_batch, tf.TensorShape([None, 6, 3]))]
        )
        tensor_list = tf.zeros([0, 3])
        for j in tf.range(seq_len):
            tf.autograph.experimental.set_loop_options(
                shape_invariants=[(tensor_list, tf.TensorShape([None, 3]))]
            )
            tiled = tf.tile(x[i][j], [dims[i][j][0], 1])
            tensor_list = tf.concat([tensor_list, tiled], axis=0)
        new_tensor = tf.pad(tensor_list, [[0,pad_sizes[i][0]], [0,0]])
        new_tensor = tf.expand_dims(new_tensor, 0)
        new_batch = tf.concat([new_batch, new_tensor], axis=0)
    return new_batch
cfrancesco

comment created time in 20 days

issue closedtensorflow/autograph

<function preprocess_time at 0x7fed4c4e9048> could not be transformed

WARNING:tensorflow:Entity <function preprocess_time at 0x7fed4c4e9048> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4

This happens at the very beginning of my runs and doesn't immediately lead to errors.:

INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/3simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1, '_save_checkpoints_steps': 10, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f5019c47160>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Not using Distribute Coordinator. INFO:tensorflow:Running training and evaluation locally (non-distributed). INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 10 or save_checkpoints_secs None. INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/3simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1, '_save_checkpoints_steps': 10, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f5019c476a0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/3simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1, '_save_checkpoints_steps': 10, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f5019c476a0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:Entity <function preprocess_time at 0x7f50693d4730> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux,export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 WARNING:tensorflow:Entity <function preprocess_time at 0x7f50693d4730> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux,export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 WARNING: Entity <function preprocess_time at 0x7f50693d4730> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux,export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4 INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

  • https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  • https://github.com/tensorflow/addons
  • https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

  • https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  • https://github.com/tensorflow/addons
  • https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From <ipython-input-15-270888898f76>:68: calling Subnetwork.new (from adanet.subnetwork.generator) with persisted_tensors is deprecated and will be removed in a future version. Instructions for updating: persisted_tensors is deprecated, please use shared instead. WARNING:tensorflow:From <ipython-input-15-270888898f76>:68: calling Subnetwork.new (from adanet.subnetwork.generator) with persisted_tensors is depr`

closed time in 20 days

le-dawg

issue commenttensorflow/autograph

<function preprocess_time at 0x7fed4c4e9048> could not be transformed

This looks like a version mismatch in gast. Newer versions will install the correct version, but 2.0 installs the newest, which is not compatible. See this comment.

Could you try either one of the following:

  • install TF 2.2, which should work with gast 0.3.3
  • install TF 2.1, which should install gast 0.2.2
  • if you wish to stay on TF 2.0, run pip install gast==0.2.2 --force-reinstall
le-dawg

comment created time in 20 days

pull request commenttensorflow/autograph

Update README.md

@jyrj Thank you for the fix!

jyrj

comment created time in 20 days

push eventtensorflow/autograph

Jayaraj J

commit sha 77a12d40c4014c3ce6978003c423629b2284b9e2

Update README.md

view details

Dan Moldovan

commit sha 305337a7d692495cbdd00c67fa9515fe3994523a

Merge pull request #7 from jyrj/master Update README.md

view details

push time in 20 days

PR merged tensorflow/autograph

Update README.md

READE.md file at https://github.com/tensorflow/autograph/tree/master/examples/sysml2019 has a minor error at line 15. pip install tensorflow is incorrectly mentioned as pip install tesnorflow which can cause the README.md to be unusable by a person who tries to reproduce by copy pasting

+1 -1

1 comment

1 changed file

jyrj

pr closed time in 20 days

issue commenttensorflow/tensorflow

Unwanted tf.function retracing when using variable-length inputs

@zaccharieramzi No fix yet. According to the code the function that the arning talks about should be cached and only traced once.

@omalleyt12 any thoughts why the tracing happens so many times?

zaccharieramzi

comment created time in 23 days

push eventtensorflow/autograph

Christian Clauss

commit sha a1af192bb2809c21c5b127907033227e4d4603b2

Python 3 compatbility via list(six.moves.xrange())

view details

Dan Moldovan

commit sha 69fdb641cc90d69a8d41d31195f6fd3033201543

Merge pull request #6 from cclauss/patch-1 Python 3 compatbility via list(six.moves.xrange())

view details

push time in 24 days

push eventtensorflow/autograph

autoih

commit sha b268d27b44a8525bedf71d010294886046561818

restore original Apache License

view details

Dan Moldovan

commit sha a9b20cdb642178b65ec45ace6356837e603efde2

Merge pull request #5 from autoih/license restore original Apache License

view details

push time in a month

PR merged tensorflow/autograph

restore original Apache License
+1 -1

0 comment

1 changed file

autoih

pr closed time in a month

push eventtensorflow/autograph

Dan Moldovan

commit sha 504ba17b389bcbf186cb317bba9c94db3cd3dfec

Delete reference_test.bzl

view details

push time in a month

push eventtensorflow/autograph

Dan Moldovan

commit sha 6fa0bd1258239c654f7465cad2ad9a72fcffd08f

Delete BUILD

view details

push time in a month

push eventtensorflow/autograph

Dan Moldovan

commit sha 8430d0bb9e8d5cfbabda5214605771903c2eef60

Minor cleanup.

view details

Dan Moldovan

commit sha eee7ab1c234ac3b1b474da8f7664e556b8b76ef6

Add reference tests.

view details

push time in a month

push eventmdanatg/Astronet-Triage

Dan Moldovan

commit sha 5e1c9b4af5099f974cc435180f9c9ccde2e42454

Save best configuration.

view details

Dan Moldovan

commit sha 42c1f4ac1209d33cfb04f8036a3b4cdf86926bb8

Add ensemble training.

view details

Dan Moldovan

commit sha c2dcfeb85cc138652eb994a1ad182a1432779c20

Fix prediction code.

view details

Dan Moldovan

commit sha 78b8d96f962aea67a49b6d303e144724800b2f9c

Add notebook to analyze results for input sweeps.

view details

push time in a month

issue commenttensorflow/tensorflow

Can we auto-convert a tensor into a numpy value when necessary in computing graph executing mode(&tf.function mode)?

@clockzhong Thank you, I agree that this is an important observation that we should document.

The difference between function arguments and values captured through the function's closure is an important one too, and is related to another known limitation in which the function is not retraced if any of the variable it closes over change.

I think both of these aspects should be mentioned in the docstring of tf.function.

clockzhong

comment created time in a month

issue commenttensorflow/tensorflow

Can we auto-convert a tensor into a numpy value when necessary in computing graph executing mode(&tf.function mode)?

Fundamentally, one cannot convert a graph tensor to numpy array because the graph does not execute in Python - so there is no NumPy at graph execution. The only way to achieve this mix is by using tf.py_function, which is ok assuming you don't want a portable model, but which will negatively affect the performance of your graph. py_function has a number of other shortcomings as well, which is the reason why it's not inserted by default into the graph.

That said, I believe the correct conversion here would be for the NumPy array to be auto-converted to Tensor. That actually happens most of the time:

  • tesor_array[numpy_index] converts numpy_index to tensor
  • tensor_x + numpy_y converts numpy_y to tensor

Therefore, the following should also be true:

  • numpy_array[tensor_index] converts numpy_array to tensor

So from that perspective, we have a bug.

So this FR could generalize as: "any operation involving mixes of numpy and tensors should consistently upcast all non-tensor values to tensor". Unfortunately, that behavior is impossible to obtain in eager mode, which is what will probably make it infeasible.

clockzhong

comment created time in a month

issue closedtensorflow/tensorflow

Simple loop is slow

<em>Please make sure that this is an issue related to performance of TensorFlow. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:performance_template</em>

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): True
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  • TensorFlow installed from (source or binary): pip
  • TensorFlow version (use command below): v2.1.0-rc2-17-ge5bf8de410
  • Python version: 3.7
  • CUDA/cuDNN version: 10.1
  • GPU model and memory: GTX 1080Ti

You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with:

  1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
  2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior Trivial loop is 300 times slower than in pure python or pytorch, note that the ratio doesn't change with the size of the loop. Note that I do get a speed up by implementing the loop using while_loop syntax, but it's far (still >250 times slower) from catching up.

Standalone code to reproduce the issue

TO BE RUN IN A NOTEBOOK

import tensorflow as tf

tf.config.set_visible_devices([], 'GPU')

@tf.function
def tf_loop(n_iter):
    val = 0.
    for i in tf.range(n_iter):
        val += 0.
    return val

def loop(n_iter):
    val = 0.
    for i in range(n_iter):
        val += 0.
    return val

n_iter = tf.constant(10000)
%timeit tf_loop(n_iter) # 92.3 ms ± 566 µs
%timeit loop(10000) # 288 µs ± 4.31 µs

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

closed time in a month

AdrienCorenflos

issue commenttensorflow/tensorflow

Simple loop is slow

See #34500 for a very similar discussion. The gist of the story is that the default TensorFlow runtime is not efficient for scalar computations. Enabling XLA compilation with @tf.function(experimental_compile=True) can greatly improve performance in cases like this. Note that the overhead of calling tf.function alone has a fixed cost of around 200 us.

AdrienCorenflos

comment created time in a month

issue commenttensorflow/tensorflow

Simple loop is slow

@AdrienCorenflos When you say trivial loop, do you mean a regular for i in range inside tf.function?

AdrienCorenflos

comment created time in a month

Pull request review commenttensorflow/tensorflow

Fix regex for scope name specification

 def testBasics(self):     self.assertEqual("bar_2", g.unique_name("bar", mark_as_used=False))     self.assertEqual("bar_2", g.unique_name("bar")) +  def testBackslashRegex(self):

Please also include a string that contains a dash character (adding it to the tests below is fine).

yongtang

comment created time in a month

Pull request review commenttensorflow/tensorflow

Fix regex for scope name specification

 def testBasics(self):     self.assertEqual("bar_2", g.unique_name("bar", mark_as_used=False))     self.assertEqual("bar_2", g.unique_name("bar")) +  def testBackslashRegex(self):+    # GitHub issue 39019, all should pass+    g = ops.Graph()+    with g.name_scope("n_CatCntc_campaign\c_campaign"):

I think we need to escape this backslash, otherwise Python will complain that "\c" is an invalid escape sequence.

yongtang

comment created time in a month

issue commenttensorflow/tensorflow

AttributeError: 'Tensor' object has no attribute '_in_graph_mode'

@helenabdr yes, and pass it as argument to the function. Although for a simple, stateless optimizerlike SGD it might not be necessary. But definitely do that with the model.

helenabdr

comment created time in a month

issue commenttensorflow/tensorflow

AttributeError: 'Tensor' object has no attribute '_in_graph_mode'

The error message did not get saved in the colab - could you copy it here? At a glance, I recommend creating the model and the optimizer outside the tf.function. You will definitely get errors because tf.function doesn't let you create variables inside it (and whenever you create a new model object, it creates a new set of variables).

helenabdr

comment created time in a month

issue closedtensorflow/tensorflow

WARNING:tensorflow:AutoGraph could not transform <function affine at 0x0000021B750C1E58> and will run it as-is.

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
  • TensorFlow version: 2.2.0-rc3
  • TensorFlow-probability version: 0.10.0-rc0
  • Python version: 3.7.6

Describe the current behavior I am using Anaconda, TensorFlow, and Spyder. When I run my script, I get the following error message:

INFO:tensorflow:Converted call: <function affine at 0x0000021B778A51F8>
    args: (<tf.Tensor 'x:0' shape=(6768,) dtype=float32>, <tf.Tensor 'kernel_diag:0' shape=(1,) dtype=float32>, <tf.Tensor 'bias:0' shape=() dtype=float32>)
    kwargs: {}

Converted call: <function affine at 0x0000021B778A51F8>
    args: (<tf.Tensor 'x:0' shape=(6768,) dtype=float32>, <tf.Tensor 'kernel_diag:0' shape=(1,) dtype=float32>, <tf.Tensor 'bias:0' shape=() dtype=float32>)
    kwargs: {}

INFO:tensorflow:Cache hit for entity <function affine at 0x0000021B778A51F8> subkey (<tensorflow.python.autograph.core.converter.ConversionOptions object at 0x0000021B790D2188>, frozenset()): _ConvertedEntityFactoryInfo(tf__affine in tmpta8nc9zk)
Cache hit for entity <function affine at 0x0000021B778A51F8> subkey (<tensorflow.python.autograph.core.converter.ConversionOptions object at 0x0000021B790D2188>, frozenset()): _ConvertedEntityFactoryInfo(tf__affine in tmpta8nc9zk)
INFO:tensorflow:Error transforming entity <function affine at 0x0000021B778A51F8>
Traceback (most recent call last):
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 538, in converted_call
    converted_f = conversion.convert(target_entity, program_ctx)
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\conversion.py", line 362, in convert
    return _instantiate(entity, converted_entity_info, free_nonglobal_var_names)
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\conversion.py", line 300, in _instantiate
    factory = converted_entity_info.get_factory()
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\conversion.py", line 94, in get_factory
    assert self.module_name in sys.modules
AssertionError
Error transforming entity <function affine at 0x0000021B778A51F8>
WARNING:tensorflow:AutoGraph could not transform <function affine at 0x0000021B778A51F8> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: 
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <function affine at 0x0000021B778A51F8> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: 
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
INFO:tensorflow:Converted call: <function affine at 0x0000021B778A51F8>
    args: (<tf.Tensor 'x:0' shape=(6768,) dtype=float32>, <tf.Tensor 'kernel_diag:0' shape=(6768,) dtype=float32>, <tf.Tensor 'bias:0' shape=(6768,) dtype=float32>)
    kwargs: {}

Converted call: <function affine at 0x0000021B778A51F8>
    args: (<tf.Tensor 'x:0' shape=(6768,) dtype=float32>, <tf.Tensor 'kernel_diag:0' shape=(6768,) dtype=float32>, <tf.Tensor 'bias:0' shape=(6768,) dtype=float32>)
    kwargs: {}

INFO:tensorflow:Whitelisted <function affine at 0x0000021B778A51F8>: from cache
Whitelisted <function affine at 0x0000021B778A51F8>: from cache
Traceback (most recent call last):
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 538, in converted_call
    converted_f = conversion.convert(target_entity, program_ctx)
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\conversion.py", line 362, in convert
    return _instantiate(entity, converted_entity_info, free_nonglobal_var_names)
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\conversion.py", line 300, in _instantiate
    factory = converted_entity_info.get_factory()
  File "C:\Users\jason\Anaconda3\lib\site-packages\tensorflow\python\autograph\impl\conversion.py", line 94, in get_factory
    assert self.module_name in sys.modules
AssertionError

Standalone code to reproduce the issue

import pandas as pd
import tensorflow as tf
# tf.autograph.set_verbosity(10,True)
import tensorflow_probability as tfp
from tensorflow_probability import distributions as tfd
# Read the data
df = pd.read_csv("https://raw.githubusercontent.com/jfhawkin/bayes_discrete_choice/master/swissmetro.dat",sep='\t')

"""Basic data and model setup"""

# Removing some observations can be done directly using pandas.
remove = (((df.PURPOSE != 1) & (df.PURPOSE != 3)) | (df.CHOICE == 0))
df.drop(df[remove].index,inplace=True)

# Definition of new variables: as tensorflow vectors
df.reset_index(inplace=True)
IDX = tf.expand_dims(tf.convert_to_tensor(df.index),1)
TRAIN_TT_SCALED = tf.expand_dims(tf.convert_to_tensor(df.TRAIN_TT / 100, dtype=tf.float32),1)
TRAIN_COST_SCALED = tf.expand_dims(tf.convert_to_tensor(df.TRAIN_CO / 100, dtype=tf.float32),1)
SM_TT_SCALED = tf.expand_dims(tf.convert_to_tensor(df.SM_TT / 100, dtype=tf.float32),1)
SM_COST_SCALED = tf.expand_dims(tf.convert_to_tensor(df.SM_CO / 100, dtype=tf.float32),1)
CAR_TT_SCALED = tf.expand_dims(tf.convert_to_tensor(df.CAR_TT / 100, dtype=tf.float32),1)
CAR_CO_SCALED = tf.expand_dims(tf.convert_to_tensor(df.CAR_CO / 100, dtype=tf.float32),1)

DATA = tf.concat([TRAIN_TT_SCALED,TRAIN_COST_SCALED,SM_TT_SCALED,SM_COST_SCALED,CAR_TT_SCALED,CAR_CO_SCALED], axis=1)

# Definition of new variables for availability, etc.: as tensorflow vectors
CHOICE = tf.convert_to_tensor(pd.get_dummies(df.CHOICE), dtype=tf.float32)
CAR_AV_SP =  tf.expand_dims(tf.convert_to_tensor(df.CAR_AV, dtype=tf.float32),1)
TRAIN_AV_SP =  tf.expand_dims(tf.convert_to_tensor(df.TRAIN_AV, dtype=tf.float32),1)
SM_AV_SP =  tf.expand_dims(tf.convert_to_tensor(df.SM_AV, dtype=tf.float32),1)

AV = tf.concat([TRAIN_AV_SP, SM_AV_SP, CAR_AV_SP], axis=1)

num_idx = df.shape[0]

"""Define the model as a partial pooling hierarchical model with varying slope for the time parameter"""

@tf.function
def affine(x, kernel_diag, bias=tf.zeros([])):
  """`kernel_diag * x + bias` with broadcasting."""
  kernel_diag = tf.ones_like(x) * kernel_diag
  bias = tf.ones_like(x) * bias
  return x * kernel_diag + bias

def mmnl_func():
    adj_AV_train = (tf.ones(num_idx) - AV[:,0]) * -9999
    adj_AV_SM = (tf.ones(num_idx) - AV[:,1]) * -9999
    adj_AV_car = (tf.ones(num_idx) - AV[:,2]) * -9999

    return tfd.JointDistributionSequential([
        tfd.Normal(loc=0., scale=1e5),  # mu_b_time
        tfd.HalfCauchy(loc=0., scale=5),  # sigma_b_time
        lambda sigma_b_time,mu_b_time: tfd.MultivariateNormalDiag(  # b_time
        loc=affine(tf.ones([num_idx]), mu_b_time[..., tf.newaxis]),
        scale_identity_multiplier=sigma_b_time),
        tfd.Normal(loc=0, scale=1e5), # a_train
        tfd.Normal(loc=0, scale=1e5), # a_car
        tfd.Normal(loc=0, scale=1e5), # b_cost
        lambda b_cost,a_car,a_train,b_time: tfd.Independent(tfd.Multinomial(
          total_count=1,
          logits=tf.stack([
              affine(DATA[:,0], tf.gather(b_time, IDX[:,0], axis=-1), (a_train + b_cost * DATA[:,1] + adj_AV_train)),
              affine(DATA[:,2], tf.gather(b_time, IDX[:,0], axis=-1), (b_cost * DATA[:,3] + adj_AV_SM)),
              affine(DATA[:,4], tf.gather(b_time, IDX[:,0], axis=-1), (a_car + b_cost * DATA[:,5] + adj_AV_car))
          ], axis=1)
        ),reinterpreted_batch_ndims=1)
    ])

mmnl_func().sample()

Full code on colab: https://drive.google.com/file/d/10OBeDVNC4tN4c8Ro2SuYhw5YfGzpXnx4/view?usp=sharing

Other info / logs Full tracebook is attached. Full code implements a sampler. Final error from full code is: ValueError: Dimensions must be equal, but are 6768 and 3 for '{{node JointDistributionSequential_1/log_prob/JointDistributionSequential_1_log_prob_IndependentJointDistributionSequential_1_log_prob_Multinomial/log_prob/JointDistributionSequential_1_log_prob_Multinomial/log_prob/mul}} = Mul[T=DT_FLOAT](value, JointDistributionSequential_1/log_prob/JointDistributionSequential_1_log_prob_IndependentJointDistributionSequential_1_log_prob_Multinomial/log_prob/JointDistributionSequential_1_log_prob_Multinomial/log_prob/LogSoftmax)' with input shapes: [6768,3], [1,3,6768]. output.txt

closed time in a month

jfhawkin

issue commenttensorflow/tensorflow

WARNING:tensorflow:AutoGraph could not transform <function affine at 0x0000021B750C1E58> and will run it as-is.

Thank you. Yes, that is a different warning emitted by tf.function. https://github.com/tensorflow/tensorflow/commit/2eb9e7c2b68123002acab835711da90d35bebca3 should remove those warnings once it's included in tf-nightly. But if those continue to appear afterwards, please ping this thread or open up a new issue and we'll have a closer look.

jfhawkin

comment created time in a month

issue commenttensorflow/tensorflow

WARNING:tensorflow:AutoGraph could not transform <function affine at 0x0000021B750C1E58> and will run it as-is.

This looks like the same issue as #38691 - it's a known issue with Spyder.

@jfhawkin could you try with tf-nightly (leave the gast version that it installs - it should be 0.3.3).

jfhawkin

comment created time in a month

issue commenttensorflow/tensorflow

tf.function with input_signature slower on unseen sequence length

@dathudeptrai Consider following #38287, it will help push for a faster resolution.

guillaumekln

comment created time in a month

issue commentgoogle/tangent

The newly released tf-nightly is incompatible with 'gast=0.2.2'

@TREiop I left a comment on #99, can you see if that helps? I'm not sure that the assert needs to be raised, but it would be good to debug why it's called repeatedly and whether it's called with the same arguments or not.

TREiop

comment created time in a month

issue commentgoogle/tangent

fail to import tangent

Sorry, it's been a while, and I don't remember why I suggested that. Have a look at https://github.com/google/tangent/blob/master/tangent/utils.py - the assertion failure suggests register_shape_function is called repeatedly with the same type, and it shouldn't. It would be useful to understand why it gets called repeatedly, and if doing that is fine, just remove the assertion.

JiafengYang

comment created time in a month

issue commentgoogle/tangent

The newly released tf-nightly is incompatible with 'gast=0.2.2'

I think the simplest way is to upgrade the code of tangent to use gast 0.3. It's not terribly complicated, because only a few types of nodes change, and it's quite mechanical. This PR shows examples of how to upgrade various elements. In most cases, gast.Str, gast.Num need to change to gast.Constant. A few other nodes including gast.Name need extra arguments that can be set to None.

TREiop

comment created time in a month

issue closedtensorflow/tensorflow

WARNING:tensorflow:Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is.

What can be done to solve this warning?

Warning message

Below is the full warning:

WARNING:tensorflow:Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: module 'gast' has no attribute 'Str' WARNING: Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: module 'gast' has no attribute 'Str'

Minimum working example

The code that generated the warning was discussed in #38471 and stored in this gist.

import tensorflow as tf

def compute_length(x): 
    return tf.strings.length(x)

def check_substring(x, substring):
    return tf.strings.regex_full_match(x,substring)


def compute_palindrome(x):
    extra_split = tf.strings.bytes_split(x)
    reverse = tf.reverse(extra_split,[0])
    reversedStr = tf.strings.reduce_join([reverse])
    return reversedStr
    
ds = tf.data.Dataset.from_tensor_slices(["Ottawa", "Stockholm", "Rabat"])

ds = ds.map(
    lambda city: (city, 
                  compute_length(city), 
                  check_substring(city, ".*lm.*"),
                  compute_palindrome(city),
                  ),
        )

num_elems = len(ds.element_spec)
for elem in ds:
   print(''.join([f"{elem[i]}" for i in range(num_elems)]))

Environment

  • python 3.7.4
  • tensorflow-gpu 2.0.0
  • tensorflow-datasets 1.3.0
  • gast 0.2.2

Running on Windows 10 under conda 4.8.3.

closed time in a month

laghaout

issue commenttensorflow/tensorflow

WARNING:tensorflow:Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is.

Actually, this should already fixed in tf-nightly. The fix will be released in TF 2.3.

Closing the issue for now, but please re-open if you still see the warnings with tf-nightly.

laghaout

comment created time in a month

issue commenttensorflow/tensorflow

WARNING:tensorflow:Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is.

Thank you, that helped! It seems that Spyder may decide to unload modules. Will send a fix.

laghaout

comment created time in a month

issue commenttensorflow/tensorflow

tf.function with input_signature slower on unseen sequence length

I think saved graphs don't need tf.function. @k-w-w thoughts?

guillaumekln

comment created time in a month

issue commenttensorflow/tensorflow

tf.function with input_signature slower on unseen sequence length

@dathudeptrai does the model warm up faster if you use tf.function(experimental_relax_shapes=True)?

guillaumekln

comment created time in a month

issue commenttensorflow/tensorflow

WARNING:tensorflow:Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is.

@laghaout Thank you. The log doesn't seem to contain any errors - did the warning appear while collecting it?

laghaout

comment created time in a month

issue commenttensorflow/tensorflow

Gradient not registered in autograph mode with tf.data.Dataset loop

@AdrienCorenflos for data in iter(dataset): will generate gradients as expected. With iter, autograph will generate code very similar to what you wrote. Without, it uses reduce which blocks gradients. We'll look into making them behave consistently.

AdrienCorenflos

comment created time in a month

issue commenttensorflow/tensorflow

WARNING:tensorflow:Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is.

Thanks for checking. So long as you don't see any module 'gast' has no attribute "foo", then the version of gast is correct. So we must look for a different cause.

To get a bit more information, could you set this env var: export AUTOGRAPH_VERBOSITY=3 and replicate the warning messages? That should generate a lot of log output that will be useful to tell what's going on.

laghaout

comment created time in a month

issue commenttensorflow/tensorflow

tf.function decorated functions fail in graph mode if any of the branches of conditionals would be invalid at runtime

This is a known inconsistency that we hope to remedy, so I'll leave the issue open to track.

The main issue is that functions like tf.size, tf.shape and tf.rank always return a Tensor, forcing AutoGraph to create a tf.cond which triggers shape checks in both tensor[1] and tensor[0]. If the shape of tensor is unknown, the code once more works correctly.

Including a few workarounds below.

If the tensor is a vector, len will work consistently:

    if len(tensor) == 2:
        return tensor[1]
    else:
        return tensor[0]

If the tensor is higher-dimensional and you indeed need the entire size (rather than just the size of dimension zero), please use .shape.num_elements() instead of tf.size until this is resolved:

    if tensor.shape.num_elements() == 2:
        return tensor[1]
    else:
        return tensor[0]

In case tensor.shape may be unknown, the following code is more complete:

def proper_size(tensor):
  if tensor.shape.is_fully_defined():
    return tensor.shape.num_elements()
  return tf.shape(tensor)

...

if proper_size(tensor):
  return tensor[1]
else:
  return tensor[0]
csachs

comment created time in a month

issue commenttensorflow/tensorflow

WARNING:tensorflow:Entity <function <lambda> at 0x000002343DCF24C8> could not be transformed and will be executed as-is.

The following versions should work (note that pip install should pull the correct dependencies in TF 2.2 and later):

TF version gast version
2.0, 2.1, 2.2 0.2.2
2.3, nightly 0.3.3

To make sure you have the correct version installed, please try force-installing it: pip install gast==<version> --force-reinstall.

laghaout

comment created time in a month

Pull request review commenttensorflow/tensorflow

Add builtin len() support of tf.data.Dataset for autograph

 def test_len(self):       tl = py_builtins.len_(data_structures.tf_tensor_list_new([3, 4, 5]))       self.assertEqual(self.evaluate(tl), 3) +  def test_len_dataset(self):

Sorry, I missed an issue that is blocking internal tests - these can also run with eager mode off. To fix that, could you modify the checks as follows:

self.assertEqual(self.evaluate(py_builtins.len_(dataset)), 3) self.assertEqual(self.evaluate(test_fn()), 3)

and so on

That way they'll be treated consistently in both eager and graph modes.

yongtang

comment created time in a month

issue commenttensorflow/tensorflow

WARNING:tensorflow:AutoGraph could not transform and will run it as-is. Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.

@nsssss

The warning will not hurt performance, but I'm curious as to what the cause might be. Can you add this line just before importing tensorflow: tf.autograph.set_verbosity(3, True)? That will print additional details with the warning message that we could use to investigate.

nsssss

comment created time in a month

issue commenttensorflow/tensorflow

OperatorNotAllowedInGraphError error in Tensorflow 2.0

@nikhilrana015 please use the updated tutorial at https://www.tensorflow.org/tutorials/generative/style_transfer. The notebook you linked uses a method of extracting intermediate layers that's unsupported in eager execution (model.get_layer(i).get_output_at(1)).

cc @omalleyt12

payne4handsome

comment created time in a month

issue commenttensorflow/tensorflow

Unwanted tf.function retracing when using variable-length inputs

@omalleyt12 @fchollet

zaccharieramzi

comment created time in a month

Pull request review commenttensorflow/tensorflow

Add builtin len() support of tf.data.Dataset for autograph

 def test_len(self):       tl = py_builtins.len_(data_structures.tf_tensor_list_new([3, 4, 5]))       self.assertEqual(self.evaluate(tl), 3) +  def test_len_dataset(self):+    dataset = dataset_ops.DatasetV2.from_tensor_slices([3, 2, 1])+    self.assertEqual(py_builtins.len_(dataset), 3)+    with self.cached_session() as sess:+      t = py_builtins.len_(dataset)+      self.assertAllEqual(self.evaluate(t), 3)++  def test_len_dataset_infinite(self):+    dataset = dataset_ops.DatasetV2.range(5).repeat().batch(2)+    with self.assertRaises(errors_impl.InvalidArgumentError):+      _ = py_builtins.len_(dataset)+    with self.cached_session() as sess:

I think this latter half of the test is redundant. Perhaps did you want to test it in graph mode? To do that, run it inside a tf.function.

yongtang

comment created time in a month

Pull request review commenttensorflow/tensorflow

Add builtin len() support of tf.data.Dataset for autograph

 def raise_zero_rank_error():                                raise_zero_rank_error)  +def _tf_dataset_len(s):+  l = cardinality.cardinality(s)++  def raise_infinite_cardinality_error():+    msg = "len requires non-infinite dataset"+    with ops.control_dependencies([control_flow_ops.Assert(False, [msg])]):+      return constant_op.constant(-1, dtype=dtypes.int32)++  def reduce_unknown_cardinality_dataset(d):+    return d.reduce(constant_op.constant(0, dtype=dtypes.int32), lambda x, _: x + 1)++  l = control_flow_ops.cond(l != cardinality.INFINITE, lambda: l, raise_infinite_cardinality_error)+

We should be able to make this even more efficient with something like this:

with ops.control_dependencies([control_flow_ops.Assert(l != cardinality.INFINITE, [msg])]):
  l = tf.identity(l)

That way, we avoid the extra tf.cond altogether.

yongtang

comment created time in a month

pull request commenttensorflow/tensorflow

Add builtin len() support of tf.data.Dataset for autograph

Interesting! I think in our case, a dataset with unknown length would be the equivalent of the generator in the Pyhton example. Wrapping an infinite generator into a list is indeed problematic, but I think for datasets such operation doesn't exist yet - it would be something in the lines of a dataset.stack()-like operation. So I think it would be totally fine to raise an error for unknown as well.

yongtang

comment created time in a month

pull request commenttensorflow/tensorflow

Add builtin len() support of tf.data.Dataset for autograph

This is great! Please see the note about efficiency and dealing with infinite datasets.

yongtang

comment created time in a month

Pull request review commenttensorflow/tensorflow

Add builtin len() support of tf.data.Dataset for autograph

 def raise_zero_rank_error():                                raise_zero_rank_error)  +def _tf_dataset_len(s):+  return s.reduce(constant_op.constant(0, dtype=dtypes.int32), lambda x, _: x + 1)

Please use the cardinality function instead, it can be much more efficient: https://github.com/tensorflow/tensorflow/blob/e5bf8de410005de06a7ff5393fafdf832ef1d4ad/tensorflow/python/data/experimental/ops/cardinality.py#L33 It can also protect against infinite loops (repeated datasets will cycle forever).

If the function returns INFINITE, then we should raise an error.

If it returns UNKNOWN, then we have two options:

  • raise an error outright (safer)
  • fall back to counting with reduce, but in that case we'd need to limit the number of iterations in the eventuality that the dataset might be actually infinite
yongtang

comment created time in a month

create barnchtensorflow/tensorflow

branch : mdanatg-patch-1

created branch time in a month

issue commenttensorflow/tensorflow

Embedding layer ~10x slower in graph mode

@omalleyt12 @fchollet

mdanatg

comment created time in a month

issue commenttensorflow/tensorflow

`nan` gradient when `tf.where` is used

@mkaze that's true - nan, inf and any other special FP value will disrupt the gradient calculation.

What happens internally is that the gradients are aggregated in this fashion: 1 * <grad of branch taken> + 0 * <grad of branch not taken>. In the former case, you have 0 * inf = nan. In the latter case, you have 1 * inf = inf. I agree it's very confusing, unfortunately a naive fix would add significant overhead to gradient calculations.

Moreover, the forward calculation doesn't need to result in a nan or inf. You can also get weird results if the gradient alone is nan or inf. For example, the cube root function is defined and well-behaved everywhere, but its derivative at zero is infinite. So this will give you a nan gradient too:

a = tf.Variable(0.0)
with tf.GradientTape() as tape:
  out = tf.where(a < 1, a, tf.pow(a, 1.0/3.0))
  grads = tape.gradient(out, a)
print(grads)

I think the tf.where workaround is useful with infinite values as well, so long as the branch not taken is forced to take a gradient that can be safely multiplied by 0. For your example, it would be something like this:

dummy_safe_value = 0
safe_a = tf.where(a > 15., dummy_safe_value, a)
out = tf.where(a > 15., a, tf.math.pow(10.0, tf.math.exp(safe_a)))

I agree that it sometimes can be impractical to do, but in principle it should always be possible as long as you control the inputs to the sensitive functions - all they have to do is force finite values in all the elements that are dropped.

0x0badc0de

comment created time in a month

issue commenttensorflow/tensorflow

Unwanted tf.function retracing when using variable-length inputs

We're investigating - it seems that a newly-added warning about function retracing seems to fire more than expected.

zaccharieramzi

comment created time in a month

issue commenttensorflow/tensorflow

Decorated the call methods of tf.keras.Model subclass with @tf.function or not , result is very different

@omalleyt12 and I spoke and we believe the difference is owed to the final Softmax layer.

In general, softmax with cross-entropy loss is prone to numerical instabilities, and the usual practice is to omit the softmax activation and use tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True). Normally, Keras will replace that automatically for you, but when you wrap the code with tf.function, it can't do it, so instead you need to do it explicitly in the code.

payne4handsome

comment created time in a month

IssuesEvent

issue closedtensorflow/tensorflow

Decorated the call methods of tf.keras.Model subclass with @tf.function or not , result is very different

System information

OS Platform and Distribution Mac os (10.14.6) TensorFlow installed from (source or binary): binary TensorFlow version (use command below): 2.0.0rc1 Python version: Python 3.6.4 I implement a model with tensorflow keras just copy official tutorials. The code is shown below

import numpy as np
from tensorflow.keras.layers import Dense, Flatten, Conv2D
from tensorflow.keras import Model
import tensorflow as tf
import time
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train, x_test
y_train, y_test = y_train , y_test
x_train = x_train[..., tf.newaxis].astype(np.float64)
x_test = x_test[..., tf.newaxis].astype(np.float64)
train_ds = tf.data.Dataset.from_tensor_slices(
(x_train, y_train)).shuffle(1000).batch(256)
test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(256)

class MyModel(Model):
    def __init__(self, *args, **kwargs):
        super(MyModel, self).__init__(args, kwargs)
        self.conv1 = Conv2D(32, 3, activation='relu')
        self.flatten = Flatten()
        self.d1 = Dense(128, activation='relu')
        self.d2 = Dense(10, activation='softmax')
        
    def call(self, x):
        x = self.conv1(x)
        x = self.flatten(x)
        x = self.d1(x)
        return self.d2(x)
model = MyModel()
model.build((512, 28, 28, 1))
loss_object = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.Adam()
train_loss = tf.keras.metrics.Mean(name='train_loss')
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')

test_loss = tf.keras.metrics.Mean(name='test_loss')
test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')
@tf.function
def train_step(images, labels):
    with tf.GradientTape() as tape:
        predictions = model(images)
        loss = loss_object(labels, predictions)
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

    train_loss(loss)
    train_accuracy(labels, predictions)
@tf.function
def test_step(images, labels):
    predictions = model(images)
    t_loss = loss_object(labels, predictions)

    test_loss(t_loss)
    test_accuracy(labels, predictions)

EPOCHS = 100

for epoch in range(EPOCHS):
    start = time.time()
    for images, labels in train_ds:
        train_step(tf.cast(images, tf.float32), labels)

    model.reset_metrics()
    for test_images, test_labels in test_ds:
        test_step(tf.cast(test_images, tf.float32), test_labels)
    end = time.time()
    template = 'Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, Test Accuracy: {}, cost:{}'
    print(
        template.format(epoch + 1, train_loss.result(),
                        train_accuracy.result() * 100, test_loss.result(),
                        test_accuracy.result() * 100, end -start))

The result show below

Epoch 1, Loss: 3.311713457107544, Accuracy: 89.99500274658203, Test Loss: 0.22570940852165222, Test Accuracy: 95.55000305175781, cost:14.249950170516968
Epoch 2, Loss: 1.7273597717285156, Accuracy: 93.3933334350586, Test Loss: 0.18705108761787415, Test Accuracy: 96.28499603271484, cost:13.976628065109253
Epoch 3, Loss: 1.1731806993484497, Accuracy: 95.00333404541016, Test Loss: 0.1706559658050537, Test Accuracy: 96.61333465576172, cost:14.125600099563599
Epoch 4, Loss: 0.8890712261199951, Accuracy: 95.9800033569336, Test Loss: 0.1615695059299469, Test Accuracy: 96.78250122070312, cost:14.998674154281616
Epoch 5, Loss: 0.7157263159751892, Accuracy: 96.64299774169922, Test Loss: 0.15943895280361176, Test Accuracy: 96.91000366210938, cost:14.496413230895996
Epoch 6, Loss: 0.5996174216270447, Accuracy: 97.10027313232422, Test Loss: 0.15636563301086426, Test Accuracy: 97.01667022705078, cost:12.977428197860718
Epoch 7, Loss: 0.5162842273712158, Accuracy: 97.44285583496094, Test Loss: 0.15558230876922607, Test Accuracy: 97.07571411132812, cost:13.014786005020142
Epoch 8, Loss: 0.4553721249103546, Accuracy: 97.66041564941406, Test Loss: 0.1616506278514862, Test Accuracy: 97.0574951171875, cost:14.170269966125488
Epoch 9, Loss: 0.4086601734161377, Accuracy: 97.81925964355469, Test Loss: 0.16302895545959473, Test Accuracy: 97.0955581665039, cost:13.114216804504395
Epoch 10, Loss: 0.3699580430984497, Accuracy: 97.98149871826172, Test Loss: 0.16444820165634155, Test Accuracy: 97.10900115966797, cost:12.783933162689209
Epoch 11, Loss: 0.3377893567085266, Accuracy: 98.12287902832031, Test Loss: 0.1649542897939682, Test Accuracy: 97.1427230834961, cost:12.907387256622314
Epoch 12, Loss: 0.31051450967788696, Accuracy: 98.2509765625, Test Loss: 0.16677971184253693, Test Accuracy: 97.17666625976562, cost:12.56903600692749
Epoch 13, Loss: 0.28760311007499695, Accuracy: 98.35563659667969, Test Loss: 0.168002188205719, Test Accuracy: 97.21846008300781, cost:13.040626049041748
Epoch 14, Loss: 0.26825159788131714, Accuracy: 98.44261932373047, Test Loss: 0.17205245792865753, Test Accuracy: 97.22142791748047, cost:13.130248785018921
Epoch 15, Loss: 0.2515304386615753, Accuracy: 98.51310729980469, Test Loss: 0.17627017199993134, Test Accuracy: 97.23332977294922, cost:12.66043996810913
Epoch 16, Loss: 0.2367033064365387, Accuracy: 98.58124542236328, Test Loss: 0.17778924107551575, Test Accuracy: 97.26062774658203, cost:12.836369037628174
Epoch 17, Loss: 0.2236185222864151, Accuracy: 98.6434326171875, Test Loss: 0.17880629003047943, Test Accuracy: 97.2752914428711, cost:12.845327854156494
Epoch 18, Loss: 0.21163013577461243, Accuracy: 98.706298828125, Test Loss: 0.18086740374565125, Test Accuracy: 97.29389190673828, cost:13.29944920539856
Epoch 19, Loss: 0.20079432427883148, Accuracy: 98.76526641845703, Test Loss: 0.18283464014530182, Test Accuracy: 97.31105041503906, cost:12.92448115348816
Epoch 20, Loss: 0.19131030142307281, Accuracy: 98.81291198730469, Test Loss: 0.18425647914409637, Test Accuracy: 97.32350158691406, cost:12.959301948547363
.....

But I find can Decorated the call methods with call method from this.

so I add annotation @tf.function to call method with above code(The rest of the code remains unchanged), just like this

   @tf.function
    def call(self, x):
        x = self.conv1(x)
        x = self.flatten(x)
        x = self.d1(x)
        return self.d2(x)

The result show below

Epoch 1, Loss: 11.509495735168457, Accuracy: 28.50666618347168, Test Loss: 11.525819778442383, Test Accuracy: 28.400001525878906, cost:14.542086839675903
Epoch 2, Loss: 11.499279022216797, Accuracy: 28.60249900817871, Test Loss: 11.579718589782715, Test Accuracy: 28.064998626708984, cost:13.102718114852905
Epoch 3, Loss: 11.471456527709961, Accuracy: 28.78444480895996, Test Loss: 11.532645225524902, Test Accuracy: 28.369998931884766, cost:13.346691846847534
Epoch 4, Loss: 11.456918716430664, Accuracy: 28.886667251586914, Test Loss: 11.518753051757812, Test Accuracy: 28.497499465942383, cost:13.706462860107422
Epoch 5, Loss: 11.445549011230469, Accuracy: 28.961334228515625, Test Loss: 11.492920875549316, Test Accuracy: 28.6560001373291, cost:13.56296968460083
Epoch 6, Loss: 11.424226760864258, Accuracy: 29.0897216796875, Test Loss: 11.270418167114258, Test Accuracy: 30.030000686645508, cost:13.60814118385315
Epoch 7, Loss: 11.216507911682129, Accuracy: 30.37714385986328, Test Loss: 11.099614143371582, Test Accuracy: 31.09000015258789, cost:13.579961061477661
Epoch 8, Loss: 11.050827980041504, Accuracy: 31.407499313354492, Test Loss: 10.950920104980469, Test Accuracy: 32.01874923706055, cost:13.574744701385498
Epoch 9, Loss: 10.918838500976562, Accuracy: 32.227962493896484, Test Loss: 10.83558464050293, Test Accuracy: 32.7400016784668, cost:12.771528959274292
Epoch 10, Loss: 10.808745384216309, Accuracy: 32.91266632080078, Test Loss: 10.74200439453125, Test Accuracy: 33.32600021362305, cost:12.579218864440918
Epoch 11, Loss: 10.719698905944824, Accuracy: 33.46500015258789, Test Loss: 10.672236442565918, Test Accuracy: 33.759090423583984, cost:12.781628131866455
Epoch 12, Loss: 10.644211769104004, Accuracy: 33.93402862548828, Test Loss: 10.609025955200195, Test Accuracy: 34.154998779296875, cost:13.060226202011108
Epoch 13, Loss: 10.57997989654541, Accuracy: 34.33307647705078, Test Loss: 10.551304817199707, Test Accuracy: 34.51692199707031, cost:13.11635136604309
Epoch 14, Loss: 10.525317192077637, Accuracy: 34.671546936035156, Test Loss: 10.500534057617188, Test Accuracy: 34.834999084472656, cost:13.43892216682434
Epoch 15, Loss: 10.476855278015137, Accuracy: 34.972442626953125, Test Loss: 10.458395957946777, Test Accuracy: 35.099334716796875, cost:13.528913021087646
Epoch 16, Loss: 10.435961723327637, Accuracy: 35.22645950317383, Test Loss: 10.421695709228516, Test Accuracy: 35.329376220703125, cost:13.542529821395874
Epoch 17, Loss: 10.397627830505371, Accuracy: 35.46509552001953, Test Loss: 10.387943267822266, Test Accuracy: 35.541175842285156, cost:12.953328132629395
Epoch 18, Loss: 10.317543983459473, Accuracy: 35.95990753173828, Test Loss: 10.273320198059082, Test Accuracy: 36.2488899230957, cost:13.799224138259888
Epoch 19, Loss: 10.211501121520996, Accuracy: 36.61640167236328, Test Loss: 10.166360855102539, Test Accuracy: 36.910526275634766, cost:13.552597045898438
Epoch 20, Loss: 10.113394737243652, Accuracy: 37.22416687011719, Test Loss: 10.071305274963379, Test Accuracy: 37.49850082397461, cost:12.61729907989502
......

When you compare these two results, The first code is obviously better than the second. I've done many experiments, and the results are the same. The first get better acc and loss just after one epoch, but the second will iterated many time from a not so good result and finally can get a better result.

so my question are : (1) What causes these two different results? because I have added @tf.function annotation to the method train_step, so whether I add @tf.function annotation to call method or not , the result should the same. (2) tensorflow keras will run with graph model automatic,why add @tf.function annotation to the call method in this api doc

closed time in a month

payne4handsome

push eventmdanatg/Astronet-Triage

Dan Moldovan

commit sha 91f4c6728b41e97a0e3ed747b348ff79d5689019

Fix tuning hyperparams.

view details

Dan Moldovan

commit sha 6864be3cc42973129a5bdb6e4c17b3e3a02c1482

Exclude non-stars from training.

view details

Dan Moldovan

commit sha cfa52a3861e210903286373614ae82f5dc8d1a0a

Minor updates.

view details

Dan Moldovan

commit sha 932062e93073d79ab5cf386175ff49b84963b3c1

Exclude non-stars from training.

view details

push time in 2 months

issue commenttensorflow/tensorflow

tf.tile each dimension individual operation not supported

Oops, wrong link - updated.

17patelumang

comment created time in 2 months

issue commenttensorflow/tensorflow

tf.tile each dimension individual operation not supported

That said, check out the comment on #38412 for a simpler solution that users some special tricks from RaggedTensors.

17patelumang

comment created time in 2 months

issue commenttensorflow/tensorflow

tf.tile each dimension individual operation not supported

b2.nrows() should work.

17patelumang

comment created time in 2 months

issue commenttensorflow/tensorflow

tf.tile each dimension individual operation not supported

Only if it gives an error. len(b2) works if b2 is a Tensor, but currently fails if it's a RaggedTensor. Working to support both.

17patelumang

comment created time in 2 months

push eventmdanatg/Astronet-Triage

Dan Moldovan

commit sha 3d05988f91cc76172a20a7943e99933d3817861b

Ensemble less aggresssively during tuning.

view details

push time in 2 months

issue closedtensorflow/tensorflow

shape_invariants "None" for all loop variables for tf 1.15.x

OS Platform and Distribution : macOS Catalina 10.15.3

TensorFlow installed from : binary

TensorFlow version : 1.15.0

Python version: 3.7.3

When I try below

import tensorflow as tf

tf.enable_eager_execution()

def tile_nd_ragged2(a, b):
  # Need a sentinel, otherwise it's hard to give it the initial shape we need.
  # We'll drop the sentinel at the end.
  acc = tf.ragged.constant([[[]]], dtype=a.dtype)

  # Work one row at a time...
  for i1 in range(len(a.nested_row_lengths()[0])):    # Should be able to write `for a1, b1 in zip(a, b)` soon.
    a1 = a[i1]
    b1 = b[i1]
    # If the components have variable length, we can't use a TensorArray anymore,
    # so use a RaggedTensor instead.
    acc1 = tf.ragged.constant([[]], dtype=a.dtype)
    def loop_test(i2, _):
      return i2 < tf.cast(a1.nested_row_lengths()[0][i1], tf.int32)
    def loop_body(i2, acc1):
      a2 = a1[i2]
      b2 = b1[i2]
      for _ in range(len(b2)):
        acc1 = tf.concat([acc1, tf.expand_dims(a2, 0)], axis=0)
      return i2 + 1, acc1

    _, acc1 = tf.while_loop(loop_test, loop_body, [0, acc1],shape_invariants=[None, tf.TensorShape([None, None])])
    acc1 = acc1[1:]  # Drop the sentinel.
    acc = tf.concat([acc, tf.expand_dims(acc1, 0)], axis=0)    # Add the row to the final result.

  acc = acc[1:]  # Drop the sentinel.
  return acc


x = tf.ragged.constant([[[b'a1', b'a2', b'a3'], [b'b1', b'b2', b'b3'], [b'c1', b'c2','c3']], [[b'd1'], [b'e1', b'e2']]])
y = tf.ragged.constant([[[b't1', b't2', b't3'], [b'u1', b'u2'], [b'v1', b'v2']], [[b'w1'], [b'x1', b'x2', b'x3']]])
print(x)
print(y)
print(tile_nd_ragged2(x, y))

I get below output

input 
a = <tf.RaggedTensor [[[b'a1', b'a2', b'a3'], [b'b1', b'b2', b'b3'], [b'c1', b'c2', b'c3']], [[b'd1'], [b'e1', b'e2']]]>
b = <tf.RaggedTensor [[[b't1', b't2', b't3'], [b'u1', b'u2'], [b'v1', b'v2']], [[b'w1'], [b'x1', b'x2', b'x3']]]>
output = <tf.RaggedTensor [[[b'a1', b'a2', b'a3'], [b'a1', b'a2', b'a3'], [b'a1', b'a2', b'a3'], [b'b1', b'b2', b'b3'], [b'b1', b'b2', b'b3'], [b'c1', b'c2', b'c3'], [b'c1', b'c2', b'c3']], [[b'd1'], [b'e1', b'e2'], [b'e1', b'e2'], [b'e1', b'e2']]]>

But when i try to put @tf.function above the function and run exactly same way it gives below error

File "tf_3.py", line 101, in <module>
   print(tile_nd_ragged2(x, y))
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 449, in __call__
   self._initialize(args, kwds, add_initializers_to=initializer_map)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 392, in _initialize
   *args, **kwds))
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1847, in _get_concrete_function_internal_garbage_collected
   graph_function, _, _ = self._maybe_define_function(args, kwargs)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2147, in _maybe_define_function
   graph_function = self._create_graph_function(args, kwargs)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2038, in _create_graph_function
   capture_by_value=self._capture_by_value),
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
   func_outputs = python_func(*func_args, **func_kwargs)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 335, in wrapped_fn
   return weak_wrapped_fn().__wrapped__(*args, **kwds)
 File "/<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 905, in wrapper
   raise e.ag_error_metadata.to_exception(e)
TypeError: in converted code:

   tf_3.py:88 tile_nd_ragged2  *
       _, acc1 = tf.while_loop(loop_test, loop_body, [0, acc1],shape_invariants=[None, tf.TensorShape([None, None])])
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py:2675 while_loop
       back_prop=back_prop)
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/while_v2.py:87 while_loop
       list(shape_invariants), expand_composites=False)
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/nest.py:536 map_structure
       structure[0], [func(*x) for x in entries],
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/nest.py:536 <listcomp>
       structure[0], [func(*x) for x in entries],
   /<path>/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py:512 _shape_invariant_to_type_spec
       % shape)

   TypeError: Expected shape to be a TypeSpec or TensorShape, got None

Opening a feature request as per discussion in ticket - https://github.com/tensorflow/tensorflow/issues/38375

Feature : support for "None" in shape_invariants for tf 1.15.x

closed time in 2 months

17patelumang

issue commenttensorflow/tensorflow

shape_invariants "None" for all loop variables for tf 1.15.x

Update: since there will not be a TF 1.16 release, the only solution is to upgrade to TF 2.2 or newer.

17patelumang

comment created time in 2 months

issue closedtensorflow/tensorflow

Tensor array can't be read when passed as argument of tf.function decorated functions

<em>Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em>

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): true
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
  • TensorFlow installed from (source or binary): pip
  • TensorFlow version (use command below): v2.1.0-rc2-17-ge5bf8de410 2.1.0

Describe the current behavior When trying to read from a tensor array passed as an argument to a tf.function decorated function, it raises an OperatorNotAllowedError:

OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed: AutoGraph did not convert this function. Try decorating it directly with @tf.function.

Describe the expected behavior It should work the same way in autograph and eager mode

Standalone code to reproduce the issue

def fun(a, tensor_array):
    loss = tf.constant(0.)
    x, y = tf.constant(0.), tf.constant(0.)
    size = tensor_array.size()
    for i in tf.range(size):
        x, y = tensor_array.read(i)
        loss += a * tf.abs(y - x) # Some dummy computation.
    return loss

decorated_fun = tf.function(fun)

data = [(1., 1.)] * 10
a_tensor_array = tf.TensorArray(dtype=tf.float32, element_shape=[2,], size=0, dynamic_size=True)
a_tensor_array = a_tensor_array.unstack(data)

a_tensor = tf.constant(1.)

print(fun(a_tensor, a_tensor_array))
print(decorated_fun(a_tensor, a_tensor_array))

Full traceback:

 ---------------------------------------------------------------------------
OperatorNotAllowedInGraphError            Traceback (most recent call last)
<ipython-input-36-fafbbc1bd984> in <module>
     17 
     18 print(fun(a_tensor, a_tensor_array))
---> 19 print(decorated_fun(a_tensor, a_tensor_array))
     20 

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\def_function.py in __call__(self, *args, **kwds)
    566         xla_context.Exit()
    567     else:
--> 568       result = self._call(*args, **kwds)
    569 
    570     if tracing_count == self._get_tracing_count():

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\def_function.py in _call(self, *args, **kwds)
    613       # This is the first call of __call__, so we have to initialize.
    614       initializers = []
--> 615       self._initialize(args, kwds, add_initializers_to=initializers)
    616     finally:
    617       # At this point we know that the initialization is complete (or less

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\def_function.py in _initialize(self, args, kwds, add_initializers_to)
    495     self._concrete_stateful_fn = (
    496         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 497             *args, **kwds))
    498 
    499     def invalid_creator_scope(*unused_args, **unused_kwds):

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2387       args, kwargs = None, None
   2388     with self._lock:
-> 2389       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2390     return graph_function
   2391 

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _maybe_define_function(self, args, kwargs)
   2701 
   2702       self._function_cache.missed.add(call_context_key)
-> 2703       graph_function = self._create_graph_function(args, kwargs)
   2704       self._function_cache.primary[cache_key] = graph_function
   2705       return graph_function, args, kwargs

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2591             arg_names=arg_names,
   2592             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2593             capture_by_value=self._capture_by_value),
   2594         self._function_attributes,
   2595         # Tell the ConcreteFunction to clean up its graph once it goes out of

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    976                                           converted_func)
    977 
--> 978       func_outputs = python_func(*func_args, **func_kwargs)
    979 
    980       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\def_function.py in wrapped_fn(*args, **kwds)
    437         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    438         # the function a weak reference to itself to avoid a reference cycle.
--> 439         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    440     weak_wrapped_fn = weakref.ref(wrapped_fn)
    441 

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\func_graph.py in wrapper(*args, **kwargs)
    966           except Exception as e:  # pylint:disable=broad-except
    967             if hasattr(e, "ag_error_metadata"):
--> 968               raise e.ag_error_metadata.to_exception(e)
    969             else:
    970               raise

OperatorNotAllowedInGraphError: in converted code:

    <ipython-input-34-fafbbc1bd984>:6 fun  *
        x, y = tensor_array.read(i)
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\ops\tensor_array_ops.py:1137 read
        return self._implementation.read(index, name=name)
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\ops\tensor_array_ops.py:753 read
        if index < 0:
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py:757 __bool__
        self._disallow_bool_casting()
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py:523 _disallow_bool_casting
        "using a `tf.Tensor` as a Python `bool`")
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py:510 _disallow_when_autograph_enabled
        " decorating it directly with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did not convert this function. Try decorating it directly with @tf.function.

closed time in 2 months

AdrienCorenflos
more