Ask questionsOptimizer clipvalue and clipnorm not working in Tensorflow 2.0

System information

  • Python 3 Google Compute Engine backend (GPU)

Describe the current behavior clipvalue and clipnorm in Optimizers does nothing!

Describe the expected behavior By setting clipvalue=0 or clipnorm=0 no training should occur (gradients should be 0!), but the network still trains, and if using a large learning rate, loss goes to nan.

Code to reproduce the issue image Gradient is clearly not zero since the network is getting modified at each iteration.

image Sanity check by setting lr=0 No training occurs when lr=0, as expected.


Answer questions tomerk

Hi Goldie, someone wants to know if a bug fix in 2.2 can be backported to 2.1.x. What's our policy on minor release versions?

Best, Tomer

On Sun, Mar 22, 2020 at 9:35 AM jlherren wrote:

Oh man, I've just been debugging my NaN losses for over an hour now. How could this even go unnoticed...

Will this be fixed for 2.1.x as well or is 2.2 the only way to get this working again? I can't disable eager mode because then I lose the cuDNN implementation of LSTMs.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe .


Related questions

ModuleNotFoundError: No module named 'tensorflow.contrib' hot 8
Error occurred when finalizing GeneratorDataset iterator hot 6
ModuleNotFoundError: No module named 'tensorflow.contrib'
When importing TensorFlow, error loading Hadoop
tf.keras.layers.Conv1DTranspose ?
tensorflow-gpu CUPTI errors hot 4
[TF 2.0] tf.keras.optimizers.Adam hot 4
Lossy conversion from float32 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning. hot 4
TF2.0 AutoGraph issue hot 4
Tf.Keras metrics issue hot 4
module 'tensorflow' has no attribute 'ConfigProto' hot 4
TF 2.0 'Tensor' object has no attribute 'numpy' while using .numpy() although eager execution enabled by default hot 4
ModuleNotFoundError: No module named 'tensorflow.examples.tutorials' hot 4
AttributeError: module 'tensorflow.python.framework.op_def_registry' has no attribute 'register_op_list' hot 4
tensorflow2.0 detected 'xla_gpu' , but 'gpu' expected hot 3
Github User Rank List