Ask questions[TF 2.0] Respect masking in Keras loss reduction (i.e., support an equivalent of default TF 1.0 loss reduction SUM_OVER_NONZERO_WEIGHTS)

System information

  • TensorFlow version (you are using): TF 2.0.0a0
  • Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

In TF 1.0, default loss reduction was SUM_OVER_NONZERO_WEIGHTS. For NLP with sequences on inputs, it normalized losses by number of valid elements (i.e., sum of non-padding words in input sentences).

With Keras API, default reduction SUM_OVER_BATCH_SIZE does not respect masking, so if a batch of sequences is passed on input, it is normalized by total batch size including padding (masked) elements. No reduction in Keras is available which would recover the previous TF 1.0 default SUM_OVER_NONZERO_WEIGHTS reduction.

Will this change the current api? How?

My proposal is to respect masking in the SUM_OVER_BATCH_SIZE reduction. If a mask is set, then BATCH_SIZE should probably correspond to the number of unmasked elements anyway.

Alternatively, a new reduction SUM_OVER_MASKED_BATCH_SIZE could be added.

Who will benefit with this feature?

I think anyone using masked losses, especially if upgrading from TF 1.


Answer questions pavithrasv

@foxik would you like to send us a PR with this change?


Related questions

ModuleNotFoundError: No module named 'tensorflow.contrib'
Error occurred when finalizing GeneratorDataset iterator
ModuleNotFoundError: No module named 'tensorflow.contrib'
When importing TensorFlow, error loading Hadoop hot 4
The flag 'log_dir' is defined twice. hot 3
[TF 2.0] Dataset has no attribute 'make_one_shot_iterator' hot 3
Lossy conversion from float32 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning. hot 3
TF2.0 AutoGraph issue hot 3
Error loading tensorflow hot 3
AttributeError: module 'tensorflow' has no attribute 'set_random_seed' hot 3
AttributeError: module 'tensorflow' has no attribute 'Session' hot 3
No tf.lite.experimental.nn.bidirectional_dynamic_rnn ops is finded hot 3
AttributeError: module 'tensorflow' has no attribute 'app' hot 3
Incorrect Error TypeError: padded_batch() missing 1 required positional argument: 'padded_shapes' hot 3
tensorflow2.0 detected 'xla_gpu' , but 'gpu' expected hot 2
Github User Rank List