profile
viewpoint
Qianli Scott Zhu qlzh727 Google California Software Engineer @google, brain, Tensorflow Keras.

keras-team/keras 46881

Deep Learning for humans

tensorflow/addons 683

Useful extra functionality for TensorFlow 2.x maintained by SIG-addons

qlzh727/addons 0

Useful extra functionality for TensorFlow 2.0 maintained by SIG-addons

qlzh727/community 0

Stores documents used by the TensorFlow developer community

qlzh727/keras 0

Deep Learning for humans

qlzh727/models 0

Models and examples built with TensorFlow

qlzh727/tensorflow 0

Computation using data flow graphs for scalable machine learning

issue closedtensorflow/tensorflow

LSTM prediction is numerically inconsistent for the last few instances.

The predictions you get may differ slightly depending on input length and position within it. E.g., if you have 11 instances of input, you get one answer for the first 8, and a different answer for the last 3. I write "may" as it happens to me with probability around 0.4. "Slightly" means in the order of the least significant bits of the float32 mantissa.

System information

  • Yes, I have written custom code, supplied below as a reprex in R using keras.
  • Tried on two platforms, with identical results. Platform A:
  • Linux Ubuntu Ubuntu 16.04.5 LTS
  • TensorFlow version: VERSION "1.7.0"
    GIT_VERSION "v1.7.0-3-g024aecf414" COMPILER_VERSION "4.8.4"
  • Python version: 2.7.12

Platform B:

  • Linux Ubuntu 14.04.6 LTS
  • TensorFlow version: tried both VERSION "1.12.0" GIT_VERSION "v1.12.0-0-ga6d8ffae09" COMPILER_VERSION "4.8.5"
  • Python version: 2.7.6

both:

  • TensorFlow installed from binary.
  • Not a mobile device.
  • CUDA/cuDNN version: Not used.
  • GPU model and memory: Not used

Describe the current behavior If the first dimension of x is n, "row" i will get one value if 0 <= i < (n&-4), but a possibly different value for (n&-4) <= i < n. (These C++/Python style 0-based indices. For R, 1-based, it's 0 < i <= bitwAnd(n, -4) versus bitwAnd(n, -4) < i <= n.)

Describe the expected behavior Reproducible prediction from same input instance, independent of row number or input length. I use generalized "row" for a slice of a tensor with a given fixed first index, e.g., x[i,,] or pred[i,].

Code to reproduce the issue This is a reprex written in R. I'd be happy to port to other languages if that's preferable.

options(digits=8)
fake <- function(shape_) {                        # arbitrary but reproducible
   array(seq_len(prod(shape_)) %% 2.71 - 1.04, shape_)
}

library(keras)
shape <- c(30,5)
model <- keras_model_sequential() %>%
   layer_lstm(units=2, input_shape=shape) %>%
   set_weights(list(fake(c(5, 8)), fake(c(2, 8)), fake(8)))

n <- 11                                           # not a multiple of 4
x <- array(rep(fake(shape), each=n), c(n, shape)) # n copies of identical input
p <- model %>% predict(x)                         # all predictions should match
p                                                 # but last n%%4 rows differ
#>             [,1]        [,2]
#>  [1,] 0.46561426 -0.22865930
#>  [2,] 0.46561426 -0.22865930
#>  [3,] 0.46561426 -0.22865930
#>  [4,] 0.46561426 -0.22865930
#>  [5,] 0.46561426 -0.22865930
#>  [6,] 0.46561426 -0.22865930
#>  [7,] 0.46561426 -0.22865930
#>  [8,] 0.46561426 -0.22865930
#>  [9,] 0.46561423 -0.22865926
#> [10,] 0.46561423 -0.22865926
#> [11,] 0.46561423 -0.22865926
(t(p)-p[1,]) * 2**26                              # the difference is low bits
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11]
#> [1,]    0    0    0    0    0    0    0    0   -2    -2    -2
#> [2,]    0    0    0    0    0    0    0    0    3     3     3
stopifnot(t(p)==p[1,])                            # all *should* be equal
#> Error in eval(expr, envir, enclos): t(p) == p[1, ] are not all TRUE

##in contrast...
x12 <- array(rep(fake(shape), each=12), c(12, shape))
p12 <- model %>% predict(x12)
stopifnot(t(p12)==p12[1,])                         # ...all is well for n == 12

<sup>Created on 2019-07-25 by the reprex package (v0.2.1.9000)</sup>

Other info / logs

closed time in 11 hours

Quiigi

issue commenttensorflow/tensorflow

LSTM prediction is numerically inconsistent for the last few instances.

Due the numerical instability, I don't think there is anything we can address here (the diff is smaller than the normal limit when we do the tests), I am going to close this bug.

Quiigi

comment created time in 11 hours

issue commenttensorflow/tensorflow

Tutorial: TF 2.0 - Text generation with keras.layer.LSTM - Fails to Generate Sensible Text

Sorry for the long wait. I think there was a backend kernel error somewhere. I rerun the same code in tutoral with LSTM layer and TF 2.1, it was able to generate reasonable result without issue. Please verify this on ur end as well. I am closing this bug and feel free to reopen it if you still see same error.

elithrar

comment created time in 11 hours

issue closedtensorflow/tensorflow

Tutorial: TF 2.0 - Text generation with keras.layer.LSTM - Fails to Generate Sensible Text

System information

  • TensorFlow version: '2.0.0-alpha0'
  • Doc Link: https://www.tensorflow.org/alpha/tutorials/sequences/text_generation#generate_text

Describe the documentation issue

The model, after 10 epochs (the default), generates "unreadable" text. This is through running the default Colab notebook via the docs/as loaded from GitHub: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r2/tutorials/sequences/text_generation.ipynb

print(generate_text(model, start_string=u"ROMEO: "))
# Output
ROMEO: VINLNKENKJVENVJNKJ3NKJKENqUKCJUKENQENVENJVENKNQKJJNKJQNK$$zzzzzzzzzzz33YYWK3JVAYYYY3NJV3NKJVJQNUKKENJNVAJJNKKxJJNKJKKJJJNKYYYJJNUJNUKC$UJVENKEVBWENVQNKJjUSJHJJJNKDENVKENVQVQVKENU$UJHENJJNJNXQGDVxjUKAQCHENKYYYJJ3NVJVMANQNKJQKENVENVJNNJNHJQUKENXENKENV3NJNUMENSENVVENVENVANQNVJQKQVHMJJNJNJNYUKENJVJNKNVANUKEVxx3NXUKENVQVUJKEN3NKJHJUKENX$qUKENQKJJNKJNNUKENKENVNQNVJKYJJNUJHENVJHYJJVANJJKYV3NVYWQJK3KJVJQUKKCJKYNVJQKEQNKEENNKJQJJNKYV3NKJJNUQDNKJKzzxJ3NV3NQNUKENKEQNKENV3NKYVQVANCHENKJJNKYVJYAN$YY33YYVJXMANQNKENVNVAVDNVANKJQNLYVQVHIVAJKJQUKENKENKJJqUKEJNNQSKCjUKENJHIVEN3NENVUVENVNVANJNVAYYVENQVKEJV3NKJx3NKYJJxQVNVHENVANMJVYYJQNUSQ3VKENVYYVNVAQXKENNKYVJJNKJQNKENX3NJXYNVJVUKJQKKJxJQNQKKENV3NKYVANJjUJQVENNKEMx3NXJNQUVQNKENRQKENQKEXKNRJNKYYYVQNVQVOVE3NHJJXKKENNQVHENVANJKENJJNN$VQUJKHENVQCHINQEVK:JVJHYYYYWqUKEMKYVJJNQVQVKAJENJQQKKENUSK$JJHEV:NVENV3NVJNJVENVENKYVJVMAYB$YYVJJNJJNUKCUKExxx3NVAQYEYYYYY3Y3VJYYYVJMJNJKK3JJNKKYV3NJNKYVQVJQNKEKYCjUKYV3NVEVJKYYYJJJjUKCENDNzzzzRNUVENVVQLNJVAKKENKJNNEQKEJNNKYYV$

Training output w/ loss:

history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])
Epoch 1/10
172/172 [==============================] - 37s 213ms/step - loss: 2.6031
Epoch 2/10
172/172 [==============================] - 33s 190ms/step - loss: 1.9073
Epoch 3/10
172/172 [==============================] - 34s 198ms/step - loss: 1.6539
Epoch 4/10
172/172 [==============================] - 33s 194ms/step - loss: 1.5161
Epoch 5/10
172/172 [==============================] - 34s 196ms/step - loss: 1.4312
Epoch 6/10
172/172 [==============================] - 33s 189ms/step - loss: 1.3695
Epoch 7/10
172/172 [==============================] - 33s 190ms/step - loss: 1.3196
Epoch 8/10
172/172 [==============================] - 34s 199ms/step - loss: 1.2751
Epoch 9/10
172/172 [==============================] - 34s 196ms/step - loss: 1.2337
Epoch 10/10
172/172 [==============================] - 34s 196ms/step - loss: 1.1941

I've validated:

  • The source data on GCS is OK - https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt
  • The char2idx mapping is correct / reversible (as expected; no typos)
  • Reduced the temperature - predictions are still invalid (as expected; they aren't really close to what we attempted to train)
  • Ran for 30 epochs (loss: 0.6435)

Continuing to debug (far from an expert) but filing this as a heads-up.

closed time in 11 hours

elithrar

push eventqlzh727/addons

qlzh727

commit sha 63d83d2f62f08c09dbbc6d31aacf22f86b09280b

Fix lint errors.

view details

push time in 14 hours

PR opened tensorflow/addons

Reviewers
Update AttentionStateWrapper to work with Keras RNN layer

Trying to fix https://github.com/tensorflow/addons/issues/1095

+25 -5

0 comment

2 changed files

pr created time in 14 hours

push eventqlzh727/addons

qlzh727

commit sha c30ac7b64838772eb578e23406780b5ce3e6e0a1

Update AttentionStateWrapper to work with Keras.

view details

push time in 14 hours

push eventqlzh727/addons

Qianli Scott Zhu

commit sha 5e3b399de4e486d0ab10147e937ebc57612888b8

Port bug fix in TF contrib to addons. (#497) * Port bug fix in TF contrib to addons. Original change at https://github.com/tensorflow/tensorflow/commit/a913689fddb70729dbce45a2cad44f4bd0f03935. * Fix lint warning.

view details

Chen Yang

commit sha 933deb7e067c4ca15a20ca6fcec18dd925604ad8

check pass through and do the expand_dims() only if needed (#464) * check pass through and do the expand_dims() only if needed * add indent to the fixed line * merge return condition to if state

view details

Tzu-Wei Sung

commit sha 89415c750c8670cb64f8c69dcd7cdf58f6bbdb8b

add hardshrink kernel (#500) * add hardshrink kernel * make linter happy

view details

AmirHosein KazemNejad

commit sha 4022d3ac392a34e540c0f856fdec591a9eadc7d3

Fixing SequenceLoss Keras incompatibility (#503) * Fix SequenceLoss incompatibility with Keras built-in loops * Remove debugging prints * Change the attribute existence checking to use more pythonic way

view details

Guillaume Klein

commit sha f6d9f310602cb6f0b2b7f57278ed9ae283d402ae

Replace some compat.v1 APIs by their v2 equivalent (#507) * Replace some compat.v1 APIs by their v2 equivalent * Fix lint error

view details

Tzu-Wei Sung

commit sha 0dc81fa26ee38f0b04fbe2ed9a1b91bdad836e6c

Add documentation for LazyAdam (#515)

view details

Dheeraj R Reddy

commit sha 2b7f080cd8af84de4ff7b01437146b50e2569d2c

FIX: MovingAverage from_config, get_config (#510) * FIX: MovingAverage from_config, get_config * `get_config` correctly serializes the wrapped optimizer using `tf.keras.optimizers.serialize()` * `from_config` to deserialize the config correctly, including the wrapped optimizer * Wrapper optimizer can be specificed using keras optimizer strings like: adam, rmsprop, adagrad, etc. * Add tests for above new functionality * Minor refactoring in `moving_average.py` * Add docstring to MovingAverage optimizer * Add name and kwargs to docstring

view details

Guillaume Klein

commit sha b086968285088816c02a38f28852178bfbe4bb98

Port assert_like_rnncell in addons utils (#513)

view details

Zhao HG

commit sha 220fad102c3edaaee292c9337a6088ccb1c93e33

Add RAdam and LookAhead optimizers (#506)

view details

Sean Morgan

commit sha 849312a995e015a8a88e981e8ae9d8ade3ad2490

Initial setup for tensorflow subsite (#486) * Initial setup for tensorflow subsite * Move examples and rename as tutorials * Spelling and style * Update README to use the word `tutorials`

view details

Guillaume Klein

commit sha 2af9442bc0627915a87fabffc578c0fc14ba29c4

Implement alternatives to some TensorFlow private APIs (#520) * Implement alternatives to some TensorFlow private APIs * Fix tests

view details

George Sterpu

commit sha 4cc2b7056e16eb30a1e903cd9394d669403951ae

typo fix (#523)

view details

Zhao HG

commit sha a6c53b09c124ed589bbe38a83f8c1d2c9bbdc4ba

Fix group operator for RectifiedAdam (#521)

view details

Mark Daoust

commit sha fd51dc7b78659c835af923f7580244ca8ded656d

Add standard flags to build_docs.py (#518) * Add standard flags to build_docs.py * ran autoformat * Readability improvements.

view details

Tomer Kaftan

commit sha f4db0b1b23788d5b512dfd758bcef491d5a7da88

Add Addons namespace to Custom Ops (#516) * Namespaced all of the custom ops * Updated C++ namespaces to not conflict w/ TF contrib ones * Ran code reformatting tool * Port bug fix in TF contrib to addons. (#497) * Port bug fix in TF contrib to addons. Original change at https://github.com/tensorflow/tensorflow/commit/a913689fddb70729dbce45a2cad44f4bd0f03935. * Fix lint warning. * check pass through and do the expand_dims() only if needed (#464) * check pass through and do the expand_dims() only if needed * add indent to the fixed line * merge return condition to if state * add hardshrink kernel (#500) * add hardshrink kernel * make linter happy * Fixing SequenceLoss Keras incompatibility (#503) * Fix SequenceLoss incompatibility with Keras built-in loops * Remove debugging prints * Change the attribute existence checking to use more pythonic way * Replace some compat.v1 APIs by their v2 equivalent (#507) * Replace some compat.v1 APIs by their v2 equivalent * Fix lint error * Add documentation for LazyAdam (#515) * Updated hardshrink custom ops & made #ifdef names more consistent. * Fix to undef

view details

prcvih

commit sha d92e084ce3a4c650db1be5094f340e6d5911f088

more efficient operation (#525)

view details

who who who

commit sha 806403525ec2880f8b0f320fb3cb80c1f94e0ca2

Fix optical_flow test case (#527)

view details

who who who

commit sha 33baec04cf3a71f41bd3007cbc1738744f1807f4

Add tanhshrink (#493) * Add tanhshrink * Update readme * Add name * Add gradient * Fix build bug * Migrate api style * Fix docs * Remove name argument * Remove name scope

view details

PyExtreme

commit sha 0e9ba6188cd3e0b90684100ff2b1bf9c7ae096d9

Add docstring correlation cost (#514) * Add Docstring CorrelationCost * Add CorrelationCost Documentation * Small reformat

view details

Tzu-Wei Sung

commit sha 370cc2aa3e047d5f83ae73c4ddd383b63f9dbf5a

fix permission (#528)

view details

push time in 15 hours

PR merged qlzh727/addons

Rebase the against upstream
+26953 -10571

0 comment

341 changed files

qlzh727

pr closed time in 15 hours

PR opened qlzh727/addons

Rebase the against upstream
+26953 -10571

0 comment

341 changed files

pr created time in 15 hours

issue commenttensorflow/addons

AttentionWrapper incompatible with RNN

Indeed, it seems that there are more assumptions in the keras backend RNN code that it will convert the state into tuples and pass it to the cell.call(). Changing that might require a lot of work, and could potentially break others. Let me see if we can reconstruct the AttentionWrapperState with the value from tuple.

georgesterpu

comment created time in 15 hours

issue commenttensorflow/tensorflow

Named dictionary inputs and outputs for tf.keras.Model

Thanks for the suggestion. Since this feature is affecting the existing API contract (type of the return value), I am not sure if we can easily change it in TF2 without breaking other users. Having said that, I think the proposed API is more intuitive, and we probably should consider it.

Adding @fchollet and @omalleyt12 here.

huyng

comment created time in a day

pull request commenttensorflow/tensorflow

Pass full input_shape to cell.build()

I'm guessing regardless of who oversees this 'contract', the process for possibly changing it will take months - so instead, how about this: pass full input_shape to both, build() and call() of all recurrent layers. Would that work?

All the recurrent layer IS currently getting full input_shape for both build() and call(). All the recurrent cells is getting input shapes without the timestep dimension. I don't think this contract need to be changed for RNN.

My question, however, is why the cell need timestep in the build() method. A cell should handle the input one timestep by one timestep, and it shouldn't care how many timesteps there will be for the whole sequence. Can u be more specific about why the timestep is needed? (Is it related to https://github.com/tensorflow/tensorflow/issues/36559?)

OverLordGoldDragon

comment created time in a day

pull request commenttensorflow/tensorflow

Pass full input_shape to cell.build()

I am not sure what you mean. In LSTM v2, the call() and build() will get same input_shape. Could u be more specific about the different shape here?

OverLordGoldDragon

comment created time in a day

issue commenttensorflow/tensorflow

How to efficiently update a tensor slice?

Oh, sorry to misunderstand your question.

First, the return value from the slide op is a Tensor, which is not the original variable.

import tensorflow as tf

a = tf.Variable(range(10), shape=(10,), name="a")
b = a[0:1]
print(b)
# tf.Tensor([0], shape=(1,), dtype=int32)

With a tensor, it doesn't have methods like "assign" or "assign_add" etc.

I think the most feasible way to create a TensorArray, which contains your slides of mean/variance value, and read/write to it within the call() body for cell. Once the layer goes over all the timesteps, you can do a stack for the TensorArray and assign the value back to the variable itself. You don't have to continuously write to the variable when processing the timesteps, since timestep t shouldn't affect the result in t+1 (if I understand you problem correctly)

OverLordGoldDragon

comment created time in 2 days

issue closedtensorflow/tensorflow

LSTM return_state=True fail with tf.keras.Sequencial model

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 10.13.6
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): v2.1.0-rc2-17-ge5bf8de410 2.1.0
  • Python version: v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14

Describe the current behavior

The call method of a tf.keras.Sequential object fails and throws an error when one layer is an instance of the tf.keras.layers.LSTM class constructed with return_state=True. Given the error message, I believe it is because the output of the call method of such LSTM layer is a list instead of a Tensor, and the call method of Sequential does not know what to do with a list.

Describe the expected behavior

I think that the call method of Sequential should know that the Tensor output of LSTM is the first element of the list when return_state=True.

Code to reproduce the issue Setting :

import tensorflow as tf
import numpy as np

print('Using Tensorflow version {} (git version {})'.format(tf.version.VERSION, tf.version.GIT_VERSION))

batch_size = 3
ts = 9
input_dim = 2
nump = np.arange(examples*batch_size*ts*input_dim, dtype=np.float32).reshape(batch_size, ts, input_dim)
dataset = tf.data.Dataset.from_tensor_slices(nump).batch(batch_size)
for x in dataset:
    print(x.shape)
return_state = True

Output:

Using Tensorflow version 2.1.0 (git version v2.1.0-rc2-17-ge5bf8de410)
(3, 9, 2)

Error with Sequential:

model_seq = tf.keras.Sequential([tf.keras.layers.LSTM(3, return_state=return_state)])
for x in dataset:
    print(model_seq(x))

Output:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-57-5500870ab2fc> in <module>
      1 model_seq = tf.keras.Sequential([tf.keras.layers.LSTM(3, return_state=return_state)])
      2 for x in dataset:
----> 3     print(model_seq(x))

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/engine/sequential.py in call(self, inputs, training, mask)
    283       # `outputs` will be the inputs to the next layer.
    284       inputs = outputs
--> 285       mask = outputs._keras_mask
    286 
    287     return outputs

AttributeError: 'list' object has no attribute '_keras_mask'

It works when constructing the model with the Functional API:

def lstm_model(return_state, ts, input_dim):
    inp = tf.keras.Input(shape=(ts, input_dim))
    out = tf.keras.layers.LSTM(3, return_state=return_state)(inp)
    return tf.keras.Model(inputs=inp, outputs=out)
    
model_func = lstm_model(return_state, ts, input_dim)

for x in dataset:
    print(model_func(x))

Output:

[<tf.Tensor: shape=(3, 3), dtype=float32, numpy=
array([[-8.8475537e-01,  2.9517543e-03, -9.9753261e-01],
       [-9.7553629e-01,  9.5521700e-06, -9.9959475e-01],
       [-9.9497062e-01,  3.0903845e-08, -9.9979442e-01]], dtype=float32)>, <tf.Tensor: shape=(3, 3), dtype=float32, numpy=
array([[-8.8475537e-01,  2.9517543e-03, -9.9753261e-01],
       [-9.7553629e-01,  9.5521700e-06, -9.9959475e-01],
       [-9.9497062e-01,  3.0903845e-08, -9.9979442e-01]], dtype=float32)>, <tf.Tensor: shape=(3, 3), dtype=float32, numpy=
array([[-7.6066346e+00,  2.9581292e-03, -3.3488092e+00],
       [-8.9999275e+00,  9.5521846e-06, -4.2520967e+00],
       [-9.0000000e+00,  3.0903848e-08, -4.5915442e+00]], dtype=float32)>]

Related question In my Functional API example, lstm_modelfails if I use inp = tf.keras.Input(shape=(ts, None)) instead of providing the explicit input dimension. The error message I get is:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-64-9b042ffca48d> in <module>
      4     return tf.keras.Model(inputs=inp, outputs=out)
      5 
----> 6 model_func = lstm_model(return_state, ts, input_dim)
      7 
      8 for x in dataset:

<ipython-input-64-9b042ffca48d> in lstm_model(return_state, ts, input_dim)
      1 def lstm_model(return_state, ts, input_dim):
      2     inp = tf.keras.Input(shape=(ts, None))
----> 3     out = tf.keras.layers.LSTM(3, return_state=return_state)(inp)
      4     return tf.keras.Model(inputs=inp, outputs=out)
      5 

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/layers/recurrent.py in __call__(self, inputs, initial_state, constants, **kwargs)
    642 
    643     if initial_state is None and constants is None:
--> 644       return super(RNN, self).__call__(inputs, **kwargs)
    645 
    646     # If any of `initial_state` or `constants` are specified and are Keras

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    746           # Build layer if applicable (if the `build` method has been
    747           # overridden).
--> 748           self._maybe_build(inputs)
    749           cast_inputs = self._maybe_cast_inputs(inputs)
    750 

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _maybe_build(self, inputs)
   2114         # operations.
   2115         with tf_utils.maybe_init_scope(self):
-> 2116           self.build(input_shapes)
   2117       # We must set self.built since user defined build functions are not
   2118       # constrained to set self.built.

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/layers/recurrent.py in build(self, input_shape)
    562     if isinstance(self.cell, Layer):
    563       if not self.cell.built:
--> 564         self.cell.build(step_input_shape)
    565 
    566     # set or validate state_spec

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/utils/tf_utils.py in wrapper(instance, input_shape)
    304     if input_shape is not None:
    305       input_shape = convert_shapes(input_shape, to_tuples=True)
--> 306     output_shape = fn(instance, input_shape)
    307     # Return shapes from `fn` as TensorShapes.
    308     if output_shape is not None:

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/layers/recurrent.py in build(self, input_shape)
   2299         regularizer=self.kernel_regularizer,
   2300         constraint=self.kernel_constraint,
-> 2301         caching_device=default_caching_device)
   2302     self.recurrent_kernel = self.add_weight(
   2303         shape=(self.units, self.units * 4),

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in add_weight(self, name, shape, dtype, initializer, regularizer, trainable, constraint, partitioner, use_resource, synchronization, aggregation, **kwargs)
    444         synchronization=synchronization,
    445         aggregation=aggregation,
--> 446         caching_device=caching_device)
    447     backend.track_variable(variable)
    448 

~/path/to/python3.6/site-packages/tensorflow_core/python/training/tracking/base.py in _add_variable_with_custom_getter(self, name, shape, dtype, initializer, getter, overwrite, **kwargs_for_getter)
    742         dtype=dtype,
    743         initializer=initializer,
--> 744         **kwargs_for_getter)
    745 
    746     # If we set an initializer and the variable processed it, tracking will not

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in make_variable(name, shape, dtype, initializer, trainable, caching_device, validate_shape, constraint, use_resource, collections, synchronization, aggregation, partitioner)
    140       synchronization=synchronization,
    141       aggregation=aggregation,
--> 142       shape=variable_shape if variable_shape else None)
    143 
    144 

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/variables.py in __call__(cls, *args, **kwargs)
    256   def __call__(cls, *args, **kwargs):
    257     if cls is VariableV1:
--> 258       return cls._variable_v1_call(*args, **kwargs)
    259     elif cls is Variable:
    260       return cls._variable_v2_call(*args, **kwargs)

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/variables.py in _variable_v1_call(cls, initial_value, trainable, collections, validate_shape, caching_device, name, variable_def, dtype, expected_shape, import_scope, constraint, use_resource, synchronization, aggregation, shape)
    217         synchronization=synchronization,
    218         aggregation=aggregation,
--> 219         shape=shape)
    220 
    221   def _variable_v2_call(cls,

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/variables.py in <lambda>(**kwargs)
    195                         shape=None):
    196     """Call on Variable class. Useful to force the signature."""
--> 197     previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
    198     for _, getter in ops.get_default_graph()._variable_creator_stack:  # pylint: disable=protected-access
    199       previous_getter = _make_getter(getter, previous_getter)

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py in default_variable_creator(next_creator, **kwargs)
   2594         synchronization=synchronization,
   2595         aggregation=aggregation,
-> 2596         shape=shape)
   2597   else:
   2598     return variables.RefVariable(

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/variables.py in __call__(cls, *args, **kwargs)
    260       return cls._variable_v2_call(*args, **kwargs)
    261     else:
--> 262       return super(VariableMetaclass, cls).__call__(*args, **kwargs)
    263 
    264 

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py in __init__(self, initial_value, trainable, collections, validate_shape, caching_device, name, dtype, variable_def, import_scope, constraint, distribute_strategy, synchronization, aggregation, shape)
   1409           aggregation=aggregation,
   1410           shape=shape,
-> 1411           distribute_strategy=distribute_strategy)
   1412 
   1413   def _init_from_args(self,

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py in _init_from_args(self, initial_value, trainable, collections, caching_device, name, dtype, constraint, synchronization, aggregation, distribute_strategy, shape)
   1540           with ops.name_scope("Initializer"), device_context_manager(None):
   1541             initial_value = ops.convert_to_tensor(
-> 1542                 initial_value() if init_from_fn else initial_value,
   1543                 name="initial_value", dtype=dtype)
   1544           if shape is not None:

~/path/to/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in <lambda>()
    120           (type(init_ops.Initializer), type(init_ops_v2.Initializer))):
    121         initializer = initializer()
--> 122       init_val = lambda: initializer(shape, dtype=dtype)
    123       variable_dtype = dtype.base_dtype
    124   if use_resource is None:

~/path/to/python3.6/site-packages/tensorflow_core/python/ops/init_ops_v2.py in __call__(self, shape, dtype)
    413       scale /= max(1., fan_out)
    414     else:
--> 415       scale /= max(1., (fan_in + fan_out) / 2.)
    416     if self.distribution == "truncated_normal":
    417       # constant from scipy.stats.truncnorm.std(a=-2, b=2, loc=0., scale=1.)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Is it normal? If so, why is that?

closed time in 2 days

durandg12

pull request commenttensorflow/tensorflow

Pass full input_shape to cell.build()

This breaks the API contract that layers.build() method will get the shape of input tensor, which should be same as the one passed in when call() is invoked. So far the cell only rely on the last dim of the inputs to build the weights, but this might not be true for other customized cells, eg if the cell somehow rely on the batch size to do certain behavior, in the time_major input case, the first dim is no longer batch, but timestep if we pass the full input shape.

OverLordGoldDragon

comment created time in 2 days

pull request commenttensorflow/community

RFC: Standalone Keras Repository

If keras is going to be moved to a standalone repo, are v1 apis (tf.get_variable(), tf.variable_scope(), etc.) for building static graph still available in the tensorflow repo? Can we build custom models without using keras? The v1 apis are really very simple and flexible to use for building static graph, while the keras api forces people to use OOP style and gives a lot of pain for industrial engineers who are familiar with static graphs and functional style.

All the public API symbols will still remain available for v1.

qlzh727

comment created time in 2 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:

Updated in the list. I agree that we should cleanup these dependency before we split the repository.

qlzh727

comment created time in 3 days

push eventqlzh727/community

qlzh727

commit sha 7144521f783d8dc12b85f2f949e31978766c804e

Update the sectoin for TF to keras dependency.

view details

push time in 3 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of

Ack, thanks for the suggestion.

qlzh727

comment created time in 3 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with the design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++Pros:+1. Under the umbrella of Keras SIG, which hosts all other Keras related projects+like keras-application, KerasTuner etc.+1. Lots of existing followers on keras-team, who may not be easily migrated to +TF project.+1. Can't easily delete keras project, which already have tons of stars and +incoming reference links. Continued existence of external Keras code will create+confusion ("why is there tensorflow/keras AND keras-team/keras?").++Cons:+1. The repo isn't under the same organization as tensorflow, which makes it hard+to manage issues/PRs and references across the organization.+1. Existing issue/PR under the same org can be transferred easily, but not cross the different org. See [here](https://help.github.com/en/github/managing-your-work-on-github/transferring-an-issue-to-another-repository).++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target a stable PIP +version of tensorflow package as dependency. It could either be (preferably in+this order):+  * a stable version+  * a release candidate version+  * a `tf-nightly` with explicit version.+Using a nightly version for testing should be motivated by the usage of a API+feature not present in the stable or pre-release version.+Depend on a floating `tf-nightly` could cause CI build to be instable, which has+been observed in other repository +[like tf-addons](https://github.com/tensorflow/addons/pull/912).+* The Keras code will be mirrored to a Google-internal code repository via+Google-internal tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it points to a+new version of `tf-nightly`, if certain behavior has been changed and wasn't+caught by unit tests. We have observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger unit test coverage by Google internel+systems, when both TF and Keras code are tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`, which uses Keras base layer and model.

I think the third package u mentioned can be avoided by the LazyLoader. For any tf code that relies on Keras, it will need keras PIP package to be installed to work. Since PIP install TF will also install Keras package, so the keras code should always be there unless user manually changed it.

qlzh727

comment created time in 3 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:

Yes, the plan is to cleanup the dependency for the private API before we split. I have the sheet to track the usage of private TF API within Keras. So far the major block of them can be converted to public API (eg, tests utils, eager context related code etc.). The rest long tail will need some rewrite.

qlzh727

comment created time in 3 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.

Done. updated wording for pure python project.

qlzh727

comment created time in 3 days

push eventqlzh727/community

qlzh727

commit sha 6872453f36b5025804d6f426543189d5c0913bdf

Update wording wrt to pure python project.

view details

push time in 3 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:

Indeed, I would also like to cleanup this dependency before we split the repository.

qlzh727

comment created time in 3 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with the design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++Pros:+1. Under the umbrella of Keras SIG, which hosts all other Keras related projects+like keras-application, KerasTuner etc.+1. Lots of existing followers on keras-team, who may not be easily migrated to +TF project.+1. Can't easily delete keras project, which already have tons of stars and +incoming reference links. Continued existence of external Keras code will create+confusion ("why is there tensorflow/keras AND keras-team/keras?").++Cons:

@fchollet here.

qlzh727

comment created time in 3 days

issue commenttensorflow/addons

AttentionWrapper incompatible with RNN

Thanks for reporting the issue, the issue is caused by force converting the initial_state tuple into list during input process. The reason behind that is keras need to have all the input tensors in its first argument for the call (). Under the hood, we flat all the inputs, inital_state into list, and then get them back in call() body.

For now, you just need the default initial state (zeros), then you can skip the line for get_initial_state(). RNN layer will call cell.get_initial_state() if the initial state is not provide. In the case u need to provide a non-default initial state, then lets update the AttentionWrapperState to accept list as inputs during construction.

In the meantime, let me check the keras code and see if we can walk around the issue for having all the tensor inputs in the first arg. This assumption might not be needed anymore after recent rework.

georgesterpu

comment created time in 3 days

issue commenttensorflow/tensorflow

A possible bug in ConvRNN2D __call__

Sorry for the very late reply. I will take a look and fix the issue.

gmrhub

comment created time in 6 days

issue commenttensorflow/tensorflow

The inheriting keras.layers.Layer does not call a compute_output_shape after switching to tf.keras from keras

Sorry for the very late reply.

I don't think compute_output_shape need to be called if we can already get the output tensor from the layer itself. It doesn't make sense to call compute_output_shape() when we have layer.output available. If the model's output doesn't carry the correct static shape information, I think we should fix the layer.call() body for that.

Could u point me to the layer in your code that produce the unexpected output?

kiflowb777

comment created time in 6 days

issue closedtensorflow/tensorflow

Unexpected behavior calling tf.keras.Model.call() with named parameters

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04.3 LTS
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.0.2
  • Python version: Python 3.7.4 Anaconda
  • CUDA/cuDNN version: ROCM:
Package: rocm-libs
Version: 3.0.6
Priority: optional
Section: devel
Maintainer: Advanced Micro Devices Inc.
Installed-Size: 13.3 kB
Depends: rocfft, rocrand, rocblas, hipblas, rocsparse, hipsparse, rocalution, rocprim, rocthrust, hipcub
Homepage: https://github.com/RadeonOpenCompute/ROCm
Download-Size: 802 B
APT-Manual-Installed: yes
APT-Sources: http://repo.radeon.com/rocm/apt/debian xenial/main amd64 Packages
Description: Radeon Open Compute (ROCm) Runtime software stack
  • GPU model and memory: AMD Radeon VII

The full environment script does not work for my machine, but:

`python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"`
>>> v2.0.0-rocm-3-g0826c3a 2.0.2

Describe the current behavior

I am getting odd behavior when calling a tf.keras.Model's call method when using the names of the method's parameters. The method works as expected when using position only arguments, but breaks when using the names. However, when I call my model_instance.call() with the names of the method parameters, things work as expected. It is making me wonder which __call__ method I am calling when simply running model_instance().

Describe the expected behavior

Using the names of the parameters in a tf.keras.Model's call method should not be raising an error.

Code to reproduce the issue

First, a little bit of setup showing that calling a tf.keras.layers.Attention instance from a function works with and without using the names of the positional arguments in a user defined function, call:

import tensorflow as tf


def call(q, v, k, mask_q=None, mask_v=None):
    """ Call attention instance """
    return attn(inputs=[q, v, k], mask=[mask_q, mask_v])

x = tf.random.uniform((1, 2, 2))
attn = tf.keras.layers.Attention(use_scale=True)
# position arguments work well
call(x, x, x)
>>> <tf.Tensor: id=89, shape=(1, 2, 2), dtype=float32, numpy=
array([[[0.62968266, 0.6612503 ],
        [0.6235384 , 0.73767066]]], dtype=float32)>
# naming the parameters also fine here
call(q=x, v=x, k=x)
>>> <tf.Tensor: id=89, shape=(1, 2, 2), dtype=float32, numpy=
array([[[0.62968266, 0.6612503 ],
        [0.6235384 , 0.73767066]]], dtype=float32)>

Things start getting weird when doing something similar within a tf.keras.Model:

class MyAttention(tf.keras.Model):
    
    def __init__(self):
        super(MyAttention, self).__init__()
        self.attention = tf.keras.layers.Attention(use_scale=True)
        
    def call(self, q, v, k, mask_q=None, mask_v=None):
        return self.attention(inputs=[q, v, k], mask=[mask_q, mask_v])


my_attention = MyAttention()
# Still works with positional arguments
my_attention(x, x, x)
>>> <tf.Tensor: id=106, shape=(1, 2, 2), dtype=float32, numpy=
array([[[0.62968266, 0.6612503 ],
        [0.6235384 , 0.73767066]]], dtype=float32)>
# Breaks when naming the arguments in my_attention:
my_attention(q=x, v=x, k=x)
>>> 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-5fa3b47998d9> in <module>
----> 1 needs_attention(q=x, v=x, k=x)

TypeError: __call__() missing 1 required positional argument: 'inputs'

Finally, if I explicitly call my_attention.call():

my_attention.call(q=x, v=x, k=x)
>>> <tf.Tensor: id=106, shape=(1, 2, 2), dtype=float32, numpy=
array([[[0.62968266, 0.6612503 ],
        [0.6235384 , 0.73767066]]], dtype=float32)>

Other info / logs Here is a gist to show this behavior: https://gist.github.com/yngtodd/f3bda25503a9611765ab33c1178db48c

closed time in 6 days

yngtodd

issue commenttensorflow/tensorflow

Unexpected behavior calling tf.keras.Model.call() with named parameters

Can't reproduce this error with latest tf-nightly 2.2.0.dev20200203. Probably the issue is fixed by the recent changes.

yngtodd

comment created time in 6 days

issue commenttensorflow/tensorflow

LSTM return_state=True fail with tf.keras.Sequencial model

So for Sequential model, we expect the layer within it only have one input and one output. The LSTM layer with return_states=True will cause it to return more than 1 output, which violate this rule.

I think the sequential model code need to be updated to show more explicit error for this case. We already show it if your model has the input_shape (which trigger model build under the hood), but we missed it in the deferred build case (input_shape is not provided by layers, but inferred when actual input is provided).

durandg12

comment created time in 6 days

issue closedtensorflow/tensorflow

keras.layers.LSTM does not work with model.evaluate after training

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian 9.11 (GCE DeepLearning image)
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): v2.0.0-rc2-26-g64c3d38 2.0.0
  • Python version: 3.7.3 (anaconda3)
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: V10.0.130 + 7.6.4.38
  • GPU model and memory: V100 16GB

Describe the current behavior model.evaluate raises this error after training

  File "lstm.py", line 76, in <module>
    model.evaluate([left, right], labels)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 833, in evaluate
    use_multiprocessing=use_multiprocessing)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 456, in evaluate
import typing
    sample_weight=sample_weight, steps=steps, callbacks=callbacks, **kwargs)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 444, in _model_iteration
    total_epochs=1)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 526, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
  (0) Not found:  Resource AnonymousIterator/AnonymousIterator1/N10tensorflow4data16IteratorResourceE does not exist.
	 [[node IteratorGetNext (defined at /home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
	 [[IteratorGetNext/_45]]
  (1) Not found:  Resource AnonymousIterator/AnonymousIterator1/N10tensorflow4data16IteratorResourceE does not exist.
	 [[node IteratorGetNext (defined at /home/zhaohaozeng/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_distributed_function_11390]

Function call stack:
distributed_function -> distributed_function

Describe the expected behavior model.evaluate should not raise this error after training.

Code to reproduce the issue Code and log: https://gist.github.com/matthew-z/e5848d545b60792dd84bfb9470ea541f

I tested with this machine on GCE:

  • n1-standard-4
  • 1 x NVIDIA Tesla V100
  • image: c1-deeplearning-common-cu100-20191003 (Google Deep Learning VM, common-dl-gpu)
  • installed anaconda3, CUDNN 7.6.4.38 (got some errors with the original 7.4 CUDNN in the image), tensorflow-gpu 2.0

Other info / logs The problem may not happen with CPU.

closed time in 6 days

matthew-z

issue commenttensorflow/tensorflow

keras.layers.LSTM does not work with model.evaluate after training

Just tested tested same code again with the latest tf-nigthly and it runs fine now.

matthew-z

comment created time in 6 days

issue commenttensorflow/tensorflow

Add more information about eager/graph context for Keras layer.call()

Just FYI, keras-team is working on some documents that clearly defines the API contract for all the important interfaces for Keras, and we will update the docs related very soon.

qlzh727

comment created time in 7 days

issue openedtensorflow/tensorflow

Add more information about eager/graph context for Keras layer.call()

Thank you for submitting a TensorFlow documentation issue. Per our GitHub policy, we only address code/doc bugs, performance issues, feature requests, and build/installation issues on GitHub.

The TensorFlow docs are open source! To get involved, read the documentation contributor guide: https://www.tensorflow.org/community/contribute/docs

URL(s) with the issue:

https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer#call

Description of issue (what needs changing):

Clear description

This is a key function that user will implement with they custom layer. Currently it is poorly documented, especially w.r.t the execution context of eager/graph. In TF 2.0, we are advocating eager execution by default. However, the call() body in keras is executed with graph context by default unless configured otherwise. It will raise error if user try to add print/debug related to items into the call body, eg print(eager_tensor.numpy()), etc.

Some related question raised in https://github.com/tensorflow/tensorflow/issues/27519.

Correct links

Yes

Parameters defined

Yes

Returns defined

Yes

Raises listed and defined

Yes

Usage example

No

Request visuals, if applicable

No

Submit a pull request?

No

created time in 7 days

PR closed tensorflow/tensorflow

Reviewers
Update serialization.py awaiting review cla: yes comp:keras size:XS

import dense_attention to void "ValueError: Unknown layer: Attention"

+1 -0

2 comments

1 changed file

shyustc

pr closed time in 9 days

pull request commenttensorflow/tensorflow

Update serialization.py

I think the same code has been already added in head now. Closing this PR since is no longer needed.

shyustc

comment created time in 9 days

pull request commenttensorflow/tensorflow

Do not ignore the `shuffle` parameter when workers=0

Hi @alexeyr, the code you are currently fixing only affects the v1 training behavior for Keras. The new code for training lives in training_v2.py and training_utils_v2.py.

Could u check whether your current PR is still applicable for the latest TF release?

alexeyr

comment created time in 9 days

issue commenttensorflow/tensorflow

How to efficiently update a tensor slice?

Hum, what's the error u see from the assign_sub()? I would expect it to work.

OverLordGoldDragon

comment created time in 9 days

issue commenttensorflow/tensorflow

How to efficiently update a tensor slice?

For quick update of a variable, eg adding value to a slide of it, I think you can use variable.assign_add()

eg

import tensorflow as tf

a = tf.Variable(range(10), shape=(10,), name="a")
print(a)
for i in range(10):
  add = [0] * 10
  add[i] = add[i] + 1
  a.assign_add(add)
  print(a)

### output of the print.
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 3, 3, 4, 5, 6, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 3, 4, 4, 5, 6, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 3, 4, 5, 5, 6, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 3, 4, 5, 6, 6, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 3, 4, 5, 6, 7, 7, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 3, 4, 5, 6, 7, 8, 8, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([1, 2, 3, 4, 5, 6, 7, 8, 9, 9], dtype=int32)>
<tf.Variable 'a:0' shape=(10,) dtype=int32, numpy=array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10], dtype=int32)>
OverLordGoldDragon

comment created time in 10 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with the design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++Pros:+1. Under the umbrella of Keras SIG, which hosts all other Keras related projects+like keras-application, KerasTuner etc.+1. Lots of existing followers on keras-team, who may not be easily migrated to +TF project.+1. Can't easily delete keras project, which already have tons of stars and +incoming reference links. Continued existence of external Keras code will create+confusion ("why is there tensorflow/keras AND keras-team/keras?").++Cons:+1. The repo isn't under the same organization as tensorflow, which makes it hard+to manage issues/PRs and references across the organization.+1. Existing issue/PR under the same org can be transferred easily, but not cross the different org. See [here](https://help.github.com/en/github/managing-your-work-on-github/transferring-an-issue-to-another-repository).++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target a stable PIP +version of tensorflow package as dependency. It could either be a `tf-nightly`+with explicit version, or a release candidate version, or a stable version.+Depend on a floating `tf-nightly` could cause CI build to be instable, which has+been observed in other repository +[like tf-addons](https://github.com/tensorflow/addons/pull/912).

Done. Updated.

qlzh727

comment created time in 10 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.+* Cross-boundary changes will require some extra handling since such changes +needs to be split into two or more PRs. Same for rollbacks.+* Tooling on the GitHub side (for code review, etc.) is not as good as+Google-internal tools. This may impact the develoment velocity for+Keras team members at Google.++### Best Practices, Tutorials and Examples++* The new Keras repository should have a new contribution guide about how to +setup a local test environment and iterate based on that. A similar one in +tf/addons can be used as an example.+* The existing TF docs needs to be updated to highlight that Keras code now lives+in a different repository, with a new process for sending PRs, etc.+* When filing an issue, people might need to consider where to send the issue,+e.g. is it a Keras issue or an issue caused by TF but surfaced by Keras. The +different ownership of the repository will also cause difficulties for +transferring the issue.++### User Impact++* No end-user facing change for current TF users; the split would only affect+developers, e.g. in-flight PRs during the transition period.+* For current Keras pip package users, they will get the new TF keras package +when they update their pip, which should have more features than the current +Keras-team version.++## Questions and Discussion Topics++1. Tools for issue tracking: we can't rely on Google-internal bug tracking tool +since it's not publicly visible, also if managing GitHub issues across the orgs +is hard, we might need to find some alternatives for tracking bugs/features etc.+2. OSS tests for TPU-related code. Since TPUs are not available during local+testing, the verification will have to be done when the PR is mirrored to Google's +internal systems.+3. Transition period for moving the Keras code from `tensorflow/tensorflow` to

Added a new section about existing PR and issue. We hope to cleanup the keras-team/keras first and bring any applicable PR/issue to TF keras first.

qlzh727

comment created time in 10 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.+* Cross-boundary changes will require some extra handling since such changes +needs to be split into two or more PRs. Same for rollbacks.+* Tooling on the GitHub side (for code review, etc.) is not as good as+Google-internal tools. This may impact the develoment velocity for+Keras team members at Google.++### Best Practices, Tutorials and Examples++* The new Keras repository should have a new contribution guide about how to +setup a local test environment and iterate based on that. A similar one in +tf/addons can be used as an example.+* The existing TF docs needs to be updated to highlight that Keras code now lives+in a different repository, with a new process for sending PRs, etc.+* When filing an issue, people might need to consider where to send the issue,

Done. Updated with a new sections about existing PR/issue and permission for the repo.

qlzh727

comment created time in 10 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|

Done. Updated with a new sections about existing PR/issue and permission for the repo.

qlzh727

comment created time in 10 days

push eventqlzh727/community

qlzh727

commit sha 1504e525f8819adc4e81259bffed69e97b17a187

Update the preferred PIP package version.

view details

qlzh727

commit sha f4af07b21f59f9a163287b2fd2b2d6772f9e7349

Update more details about the Github migration.

view details

push time in 10 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with the design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++Pros:+1. Under the umbrella of Keras SIG, which hosts all other Keras related projects+like keras-application, KerasTuner etc.+1. Lots of existing followers on keras-team, who may not be easily migrated to +TF project.+1. Can't easily delete keras project, which already have tons of stars and +incoming reference links. Continued existence of external Keras code will create+confusion ("why is there tensorflow/keras AND keras-team/keras?").++Cons:+1. The repo isn't under the same organization as tensorflow, which makes it hard+to manage issues/PRs and references across the organization.+1. Existing issue/PR under the same org can be transferred easily, but not cross the different org. See [here](https://help.github.com/en/github/managing-your-work-on-github/transferring-an-issue-to-another-repository).++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target a stable PIP +version of tensorflow package as dependency. It could either be a `tf-nightly`+with explicit version, or a release candidate version, or a stable version.+Depend on a floating `tf-nightly` could cause CI build to be instable, which has+been observed in other repository +[like tf-addons](https://github.com/tensorflow/addons/pull/912).+* The Keras code will be mirrored to a Google-internal code repository via+Google-internal tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it points to a

The build failure might still occur if we point to a newer version of TF, but probably caught by the presubmit test.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.+* Cross-boundary changes will require some extra handling since such changes +needs to be split into two or more PRs. Same for rollbacks.+* Tooling on the GitHub side (for code review, etc.) is not as good as+Google-internal tools. This may impact the develoment velocity for+Keras team members at Google.++### Best Practices, Tutorials and Examples++* The new Keras repository should have a new contribution guide about how to +setup a local test environment and iterate based on that. A similar one in +tf/addons can be used as an example.+* The existing TF docs needs to be updated to highlight that Keras code now lives+in a different repository, with a new process for sending PRs, etc.+* When filing an issue, people might need to consider where to send the issue,+e.g. is it a Keras issue or an issue caused by TF but surfaced by Keras. The +different ownership of the repository will also cause difficulties for +transferring the issue.++### User Impact++* No end-user facing change for current TF users; the split would only affect+developers, e.g. in-flight PRs during the transition period.+* For current Keras pip package users, they will get the new TF keras package +when they update their pip, which should have more features than the current +Keras-team version.++## Questions and Discussion Topics++1. Tools for issue tracking: we can't rely on Google-internal bug tracking tool +since it's not publicly visible, also if managing GitHub issues across the orgs +is hard, we might need to find some alternatives for tracking bugs/features etc.+2. OSS tests for TPU-related code. Since TPUs are not available during local+testing, the verification will have to be done when the PR is mirrored to Google's +internal systems.+3. Transition period for moving the Keras code from `tensorflow/tensorflow` to+`keras-team/keras`. All in-flight PRs / issues will be affected: they need

We want to minimize the period of transition, since during the period, there will be kind of two source of truth, which might confuse user which one to use/contribute. I will update this RFC with more details about the PR/issue transition.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.+* Cross-boundary changes will require some extra handling since such changes +needs to be split into two or more PRs. Same for rollbacks.+* Tooling on the GitHub side (for code review, etc.) is not as good as+Google-internal tools. This may impact the develoment velocity for+Keras team members at Google.++### Best Practices, Tutorials and Examples++* The new Keras repository should have a new contribution guide about how to +setup a local test environment and iterate based on that. A similar one in +tf/addons can be used as an example.+* The existing TF docs needs to be updated to highlight that Keras code now lives+in a different repository, with a new process for sending PRs, etc.+* When filing an issue, people might need to consider where to send the issue,+e.g. is it a Keras issue or an issue caused by TF but surfaced by Keras. The +different ownership of the repository will also cause difficulties for +transferring the issue.++### User Impact++* No end-user facing change for current TF users; the split would only affect+developers, e.g. in-flight PRs during the transition period.+* For current Keras pip package users, they will get the new TF keras package +when they update their pip, which should have more features than the current +Keras-team version.++## Questions and Discussion Topics++1. Tools for issue tracking: we can't rely on Google-internal bug tracking tool +since it's not publicly visible, also if managing GitHub issues across the orgs +is hard, we might need to find some alternatives for tracking bugs/features etc.

Yes, this is mainly refer to the transferring the issue across the org.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.+* Cross-boundary changes will require some extra handling since such changes +needs to be split into two or more PRs. Same for rollbacks.+* Tooling on the GitHub side (for code review, etc.) is not as good as+Google-internal tools. This may impact the develoment velocity for+Keras team members at Google.++### Best Practices, Tutorials and Examples++* The new Keras repository should have a new contribution guide about how to +setup a local test environment and iterate based on that. A similar one in +tf/addons can be used as an example.+* The existing TF docs needs to be updated to highlight that Keras code now lives+in a different repository, with a new process for sending PRs, etc.+* When filing an issue, people might need to consider where to send the issue,+e.g. is it a Keras issue or an issue caused by TF but surfaced by Keras. The +different ownership of the repository will also cause difficulties for +transferring the issue.++### User Impact++* No end-user facing change for current TF users; the split would only affect+developers, e.g. in-flight PRs during the transition period.+* For current Keras pip package users, they will get the new TF keras package +when they update their pip, which should have more features than the current +Keras-team version.++## Questions and Discussion Topics++1. Tools for issue tracking: we can't rely on Google-internal bug tracking tool 

Thanks for the notice. I will check with infra team and support team to see if they have any suggestions for improving this situation. Will leave this open for discussion in the design meeting.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.

@gunan from TF build team, and also @seanpmorgan who mentioned this in the SIG-addon meeting.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|

Thanks all for pointing this problem out. Since we are reusing the existing keras project with totally new code base, I think we will need to cleanup any existing PR/Issue and group since many of them maybe incorrect with new code. I will update this RFC with more details.

qlzh727

comment created time in 11 days

pull request commenttensorflow/community

RFC: Standalone Keras Repository

@Dref360, could check your CLA setting?

frederic.branchaud-charron@usherbrooke.ca | need_author_cla

qlzh727

comment created time in 11 days

pull request commenttensorflow/community

RFC: Standalone Keras Repository

Will the ci still run keras tests when changes are made to the tensorflow repository ?

It will only run tests in TF repository that relies on some Keras functionality, eg base_layers. Since Keras rely on TF as a dependency, we will not run any Keras for any PR to TF.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process

Updated with 86d2c37. I will leave this open since your comment contains more details.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.

Done.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.+* Cross-boundary changes will require some extra handling since such changes 

Yes, this is intended. Having said that, we don't expect this to happen a lot, since most of the TF/keras PR only touch either side, not both at the same time.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.

Updated for the section for repo comparison. I think the we lean towards keras-team/keras since it already has a lot of uses/traffic/stars and incoming references.

qlzh727

comment created time in 11 days

push eventqlzh727/community

qlzh727

commit sha b654b44ea14f07090d841efc37690790b92bed5c

Update tf to Keras usage with more details.

view details

qlzh727

commit sha 7c57540a428d0b3c3e1fa0ac727580545396175f

Change to use the list view for comparison. Make it easy for comment to point to the exact place.

view details

qlzh727

commit sha 86d2c3737ba5f285e486494196a3de17383148a7

Update the section for PIP package dependency. Intake the suggestion from @gabrieldemarmiesse based on the tf-addons.

view details

push time in 11 days

push eventqlzh727/community

Qianli Scott Zhu

commit sha 0109f0f81f9920f6e11c608d2e89e63a95f184e1

Update rfcs/20200205-standalone-keras-repository.md Co-Authored-By: Gabriel de Marmiesse <gabrieldemarmiesse@gmail.com>

view details

push time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|

Ack, will put them as list, which makes it easy to comment.

qlzh727

comment created time in 11 days

push eventqlzh727/community

Qianli Scott Zhu

commit sha 3f0dc9276eb4f287120aa3ac60ee37fd605098d6

Update rfcs/20200205-standalone-keras-repository.md Co-Authored-By: Frédéric Branchaud-Charron <frederic.branchaud-charron@usherbrooke.ca>

view details

push time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the 

Ack, assuming this is just a generic comment and doesn't need to be addressed.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies++The TensorFlow pip package will auto-install the Keras package, which shouldn't make +any difference on the end-user side. Under the hood, Keras will be a different +package imported by `tf_core`, like what we do for TF estimator.++### Developer experience Impact++* The local build and test times should be greatly reduced, since compiling TF+is no longer needed, and Keras is a pure-Python project.

@gabrieldemarmiesse, I think at this stage, we can't remove bazel and use pytest since it will affect too much of the Google internal builds (we rely on bazel heavily internally). Wrt to CI choice, I think we are open for that, and I will take a look how github workflow action is setup for other SIGs.

@seanpmorgan, for tf-addons, how is your current experience with Github workflow actions? Does it meet all the need? eg showing build history, GPU testing, presubmit notification, etc?

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process

Thanks @gabrieldemarmiesse for the detailed suggestion. I do like the idea of pinning to a certain version which should greatly reduce the chance of breakage. I think pinning the version (nightly or stable) is a very good alternative for reproducibility and supporting the rapid development.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies+

Yes, I had offline discussion with Anna before, and we would prefer keras instead of Keras. I think when Anna wrote her design, she was not sure about what package we will use, and she assumed we will use the similar name as tensorflow_estimator, which is why she put tensorflow_keas in her design. @annarev for #182.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies+

@gabrieldemarmiesse, yes. tf.keras symbol will still work, and so as keras. Under the hood, tf.keras will load the python symbol from keras.

qlzh727

comment created time in 11 days

Pull request review commenttensorflow/community

RFC: Standalone Keras Repository

+# Standalone Keras Repository++| Status        | Proposed |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [202](https://github.com/tensorflow/community/pull/202) |+| **Author(s)** | Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com) |+| **Sponsor**   | Karmel Allison (karmel@google.com) |+| **Updated**   | 2020-02-05                         |++## Objective++Move the Keras code from the TensorFlow main GitHub repository to its own+repository, with TensorFlow as a dependency.++## Motivation++### Build times++Building the open-source TensorFlow project end-to-end is an extensive exercise. +With a standard GCP instance, it might take more than one hour to finish the whole+build process (it might take longer with a Mac laptop). Although the local build +cache might help speed up the follow-up builds, the initial time cost is too +high for regular software development workflows. Internally, Google has a+distributed build and caching service, which Googlers heavily rely on,+that can build TensorFlow and run all Keras tests within 5 mins. Sadly,+we can't expose this to external contributors.++Currently, any contribution to Keras code will require building all of+TensorFlow, which is quite expensive to do for average users.+Having a separate repository will allow the Keras package to be built+without building TensorFlow. This should greatly improve the +velocity of open-source developers when they contribute to Keras code.++### Community Benefit++The difficulty of building TensorFlow from scratch in order to make a PR+to Keras code has been a significant source of issues:++* It discouraged contributions, since many external developers couldn't test+their changes and make sure they were correct.+* External developers would send unverified PRs, and Google reviewers spend time back +and forth, fixing the PR. Sometimes PR is just not moving forward because of the+lengthy feedback loop.++With the new standalone Keras repository, external contributors+should experience much shorter turn-around time when +building/testing Keras, since they don't need to build TensorFlow anymore.+This should  have a positive impact on building a vibrant open-source+developer community.++In addition, by getting the Keras team at Google to start developing Keras+using the same public tools and infrastructure as third-party developers,+we make the development process more transparent and more community-oriented.++### TensorFlow API modularity++There are other side-benefits if we split the repository. Currently, Keras+has to rely on a number of private TensorFlow APIs. However, a litmus test+of the quality of the public TensorFlow low-level APIs is that they should+be strictly sufficient to a higher-level API like Keras.+After splitting the repository, Keras will have to import TensorFlow and +rely exclusively on public APIs. If Keras still ends up using TensorFlow+private features, it  might be an indication of tight coupling of+implementation details. If certain private features are extensively used,+we might want to consider exposing them  as public low level API.++This design is also aligned with thee design for+[Modular TensorFlow](https://github.com/tensorflow/community/blob/master/rfcs/20190305-modular-tensorflow.md), +which splits the TensorFlow project into smaller components that are not+tightly coupled together.+++## Design Proposal++### New location of the code++GitHub: the code will live at [keras-team/keras](https://github.com/keras-team/keras), +joining the other Keras SIG projects and replacing the current external Keras +codebase. `tf.Keras` will also replace Keras on PyPI.++Also considered: `tensorflow/keras`.++| keras-team/keras   | tensorflow/keras |+:------------------- |:------------------------------------------- |+|Under the umbrella of Keras SIG, which hosts all other Keras related projects like keras-application, KerasTuner etc.|Under the umbrella of tensorflow, which also hosts other TF related projects.|+|Lots of existing followers on keras-team, who may not be easily migrated to TF project.|No cross org repo management cost on GitHub. Could rely on a lot of existing setup in TensorFlow.|+|Can't easily delete keras project, which already have tons of stars and incoming reference links. Continued existence of external Keras code will create confusion ("why is there tensorflow/keras AND keras-team/keras?")|Issue/PR under the same org can be transferred easily, but not cross the different org. See here|++### Source of Truth++TensorFlow uses a Google-internal code repository as its source of truth. Every PR+submitted though GitHub is converted to a Google-internal change first,+submitted through the internal system, and then copied to GitHub as commits.+At the same time, PR is marked as merged with the corresponding commit hash.++Likewise, issue tracking and code review takes place through Google-internal tools.++For Keras, since we are trying to promote community engagement, we hope to use +GitHub as source of truth. This will have the following implications:++* We expect the majority of the code development/contribution from GitHub+and the dev tools / tests / scripts should focus on the GitHub development use+case. See below for more details.+* Keras CI/presubmit build for the GitHub repo should target the `tf-nightly` pip+package as dependency. This means any change to TF will take at most 24+hours to be reflected on the Keras side.+* The Keras code will be mirrored to a Google-internal code repository via Google-internal +tools within a very short time window after each change.+The Google-internal CI tests will run on HEAD for both Keras and TF code.+* The CI build for the repository on GitHub might break when it sees a+new version of `tf-nightly`, if  certain behavior has been changed and wasn't+caught by unit tests. We have  observed a few similar cases with+[tf/addons](https://github.com/tensorflow/addons).+We hope this can be reduced by stronger+unit test coverage by Google internel systems, when both TF and Keras code are +tested at HEAD.+* pip package management. Keras will now follow the `tf-estimator` approach. +"pip install tensorflow" should also install Keras (from PyPI) as well.+There are more details for the pip package in the+[Improved pip package structure](https://github.com/tensorflow/community/pull/182) RFC.++### Dependency Cleanup++As the high-level API of TensorFlow, Keras should have a direct dependency on+TF low-level APIs, but not the other way around. Unfortunately, there is some existing reverse +logic in the TF code that relies on Keras, which we should update/remove +when we split the repository.++The current usage of Keras from TensorFlow are:+* Unit tests, which should be converted to integration tests, or port the tests+to Keras repository.+* `feature_column`.+* Legacy `tf.layers` in v1 API.+* legacy RNN cells.+* TPU support code for `optimizer_v2`.+* SavedModel.+* TF Lite.++All Keras imports in integration tests can be changed to use dynamic import like below:++```python+try:+   from tensorflow.python.keras.engine import base_layer+except ImportError:+   tf.logging.error('keras is not installed, please pip install keras')+   base_layer = None+```++### Update Keras to only use public TF APIs++The current Keras code will still work if we do e.g.:+```python+from tensorflow.python.ops import array_ops++ones = array_ops.ones([2, 3])+```++However, since Keras is a separate repository, having it only use TF +public APIs will heavily reduce the chance of breakage caused by relying+on private methods or implementation details. We think this point is+critial to the health of the project. This also allows TF to change internal +implementation details without worrying about breaking Keras.++The converted code should look like e.g.:++```python+import tensorflow as tf++ones = tf.ones([2, 3])+```++During this conversion, we might notice that certain TF features used in Keras are+not public. A decision should be made on a case-by-case basis:++* Copy the functionality from TF to Keras.+* Replace the usage with another alternative TF public API.+* Make the functionality a new TF public API.++**Note that the open-source community is encouraged to contribute to this effort.**++### Two-stage change process++For any change that is affecting both TensorFlow and Keras, the change+will need to be split into two, one as a PR to the TF repo,+and the other as a PR to the Keras repo. Here are some common scenarios:++1. Adding a new feature to TensorFlow, and having Keras rely on it. Note that the +TF change needs to be submitted first, and the Keras PR needs to wait for the new TF+nightly to become available on PyPI.++Also note that any rollback of the TF PR will cause Keras to break, the+rollback sequence should be PR 33333 and then PR 22222 (see example below).+The Google-internal test for TF should catch the error if the rollback sequence+is not correct.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# +++ tensorflow/python/ops/array_ops.py+def some_new_function(inputs):+   ...++# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     array_ops.some_new_function(inputs)+     ...+```++```python+# New scenario.+# PR 22222 (1 file updated)+# +++ tensorflow/python/ops/array_ops.py+@tf.export('some_new_function')+def some_new_function(inputs):+   ...++==================================+# PR 33333 (1 file updated)+# +++ tensorflow/python/keras/layers/core.py++class new_layer(Layer):++  def call(inputs):+     tf.some_new_function(inputs)+     ...+```++2. Changing the behavior of an existing TF API.++Note that the PR 22222 needs to be submitted with both the new and old+function since Google internal CI is still testing from HEAD.+The previous function can be +deleted after PR 33333 is submitted. Also note that this issue is caused by +Keras not using exclusively public TF API, but relying on TF implementation details.+Moving towards only using public APIs should reduce the likelihood of this kind of issue.++```python+# Existing scenario.+# PR 11111 (2 files updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+    ...+>>>+def new_function(inputs, knob1=False, knob2=1):+    ...+# tensorflow/python/keras/layers/core.py++class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```++```python+# New scenario.+# PR 22222 (1 file updated)+# tensorflow/python/ops/array_ops.py+<<<+def existing_function(inputs):+   ...+>>>+def existing_function(inputs):+  return new_function(+    inputs, +    knob1=False,+    knob2=1)++def new_function(inputs, knob1, knob2=1):+    ...++==================================+# PR 33333 (1 file updated)+# tensorflow/python/keras/layers/core.py+class existing_layer(Layer):++  def call(inputs):+<<<+    array_ops.existing_function(inputs)+     ...+>>>+    array_ops.new_function(+        inputs, +        knob1=True,+        knob2=3)+```+++### Performance Implications++There may be some performance implications as we move towards only using+public TF APIs. We need to maintain a benchmark to ensure that there+is no performance regression.++### Dependencies+

Anna's design covers more details about the PIP package structure. https://github.com/tensorflow/community/pull/182. Currently TF package is an umbrella package, and the tensorflow_core is the package that contains all the real code. The new Keras package will depend on tensorflow_core, and TF umbrella package will depend on Keras.

This the same approach used for tensorflow_estimator, which relies on Tensorflow ops, but still exposed under tf.estimator.

qlzh727

comment created time in 11 days

issue commenttensorflow/tensorflow

Custom layer go_backwards does not work

We have update the docstring with more details about the requirement for the custom layer.

Jbiloki

comment created time in 13 days

issue closedtensorflow/tensorflow

Custom layer go_backwards does not work

I am implementing a custom recurrent class that inherits tf.layers.Layer, when using the Bidirectional wrapper I get the error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-7bd5b5269810> in <module>
----> 1 a = TimeDistributed(Bidirectional(char_recurrent_cell))

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/wrappers.py in __init__(self, layer, merge_mode, weights, backward_layer, **kwargs)
    434     if backward_layer is None:
    435       self.backward_layer = self._recreate_layer_from_config(
--> 436           layer, go_backwards=True)
    437     else:
    438       self.backward_layer = backward_layer

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/wrappers.py in _recreate_layer_from_config(self, layer, go_backwards)
    493     config = layer.get_config()
    494     if go_backwards:
--> 495       config['go_backwards'] = not config['go_backwards']
    496     if 'custom_objects' in tf_inspect.getfullargspec(
    497         layer.__class__.from_config).args:

KeyError: 'go_backwards'

This is the code for the layer itself:

class RecurrentConfig(BaseLayer):
    '''Basic configurable recurrent layer'''
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.layers: List[layers.Layer] = stack_layers(self.params,
                                                       self.num_layers,
                                                       self.layer_name)

    def call(self, inputs: np.ndarray) -> layers.Layer:
        '''This function is a sequential/functional call to this layers logic
        Args:
            inputs: Array to be processed within this layer
        Returns:
            inputs processed through this layer'''
        processed = inputs
        for layer in self.layers:
            processed = layer(processed)
        return processed

    @staticmethod
    def default_params() -> Dict[Any, Any]:
        return{
            'units': 32,
            'recurrent_initializer': 'glorot_uniform',
            'dropout': 0,
            'recurrent_dropout': 0,
            'activation': None,
            'return_sequences': True
        }

I have attempted to add the go_backwards to the config that is retrieved when get_config() is called but this results in another error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-7bd5b5269810> in <module>
----> 1 a = TimeDistributed(Bidirectional(char_recurrent_cell))

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/wrappers.py in __init__(self, layer, merge_mode, weights, backward_layer, **kwargs)
    430     # Recreate the forward layer from the original layer config, so that it will
    431     # not carry over any state from the layer.
--> 432     self.forward_layer = self._recreate_layer_from_config(layer)
    433 
    434     if backward_layer is None:

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/wrappers.py in _recreate_layer_from_config(self, layer, go_backwards)
    506       return layer.__class__.from_config(config, custom_objects=custom_objects)
    507     else:
--> 508       return layer.__class__.from_config(config)
    509 
    510   @tf_utils.shape_type_conversion

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in from_config(cls, config)
    517         A layer instance.
    518     """
--> 519     return cls(**config)
    520 
    521   def compute_output_shape(self, input_shape):

~/nlpv3-general/nlp-lib/src/main/python/mosaix_py/mosaix_learn/layers/recurrent_layers.py in __init__(self, *args, **kwargs)
     12     '''Basic configurable recurrent layer'''
     13     def __init__(self, *args, **kwargs):
---> 14         super().__init__(*args, **kwargs)
     15         self.layers: List[layers.Layer] = stack_layers(self.params,
     16                                                        self.num_layers,

~/nlpv3-general/nlp-lib/src/main/python/mosaix_py/mosaix_learn/layers/base_layer.py in __init__(self, params, mode, layer_name, num_layers, cust_name, **kwargs)
     17                  cust_name: str = '',
     18                  **kwargs):
---> 19         super().__init__(params, mode, **kwargs)
     20         self.layer_name = layer_name
     21         self.cust_name = cust_name

~/nlpv3-general/nlp-lib/src/main/python/mosaix_py/mosaix_learn/configurable.py in __init__(self, params, mode, **kwargs)
     61 
     62     def __init__(self, params: Dict[AnyStr, Any], mode: ModeKeys, **kwargs):
---> 63         super().__init__(**kwargs) #type: ignore
     64         self._params = _parse_params(params, self.default_params())
     65         self._mode = mode

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    455     self._self_setattr_tracking = False  # pylint: disable=protected-access
    456     try:
--> 457       result = method(self, *args, **kwargs)
    458     finally:
    459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __init__(self, trainable, name, dtype, dynamic, **kwargs)
    184     }
    185     # Validate optional keyword arguments.
--> 186     generic_utils.validate_kwargs(kwargs, allowed_kwargs)
    187 
    188     # Mutable properties

~/opt/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/keras/utils/generic_utils.py in validate_kwargs(kwargs, allowed_kwargs, error_message)
    716   for kwarg in kwargs:
    717     if kwarg not in allowed_kwargs:
--> 718       raise TypeError(error_message, kwarg)

TypeError: ('Keyword argument not understood:', 'go_backwards')

Version info is: tf_version: '2.1.0-dev20191125' git_version: 'v1.12.1-19144-gf39f4ea3fa'

closed time in 13 days

Jbiloki

issue commenttensorflow/tensorflow

Predicting on custom model fails

The walk around so far is to update your compute_output_shape method with the wrapper of tf.TensorShape(), so that the nest lib will not try to expand it. We will fix the issue on our side in the meantime.

  def compute_output_shape(self, input_shape):
    print('[Generator] - compute_output_shape() input_shape: {}'.format(input_shape))
    return tf.TensorShape([input_shape[0], 28, 28, 1])
moono

comment created time in 13 days

issue commenttensorflow/tensorflow

Predicting on custom model fails

Digging more into details, I think the issue is caused by how we build model input/output on a dynamic model. It is using the result of model.compute_output_shape to build a symbolic output, and incorrectly use nest.map_structure, which in this case expand a 4D shape into 4 outputs. The number of wrong outputs cause the error down the stream.

https://github.com/tensorflow/tensorflow/blob/af38219818b1b677277018468c089593965d8c24/tensorflow/python/keras/engine/base_layer.py#L2219

moono

comment created time in 13 days

issue commenttensorflow/tensorflow

Custom layer go_backwards does not work

Writing the custom layer with go_backwards will take quite some work, some sample code will be like:

import tensorflow as tf

class DummyLayer(tf.keras.layers.Layer):
  def __init__(self, go_backwards=False, *args, **kwargs):
    super(DummyLayer, self).__init__(*args, **kwargs)
    self.go_backwards = go_backwards
    self.a = tf.keras.layers.LSTM(2, go_backwards=self.go_backwards)

  def call(self, inputs):
    return self.a(inputs)

  @property
  def return_sequences(self):
    return self.a.return_sequences

  @property
  def return_state(self):
    return self.a.return_state

  def get_config(self):
    config = {'go_backwards': self.go_backwards}
    base_config = super(DummyLayer, self).get_config()
    return dict(list(base_config.items()) + list(config.items()))

tf.keras.layers.Bidirectional(DummyLayer())
Jbiloki

comment created time in 13 days

issue commenttensorflow/tensorflow

Custom layer go_backwards does not work

Hi @Jbiloki.

Thanks for reporting the issue. For any layer that need to work with Bidirectional wrapper, it need to support processing the inputs in a reversed order. For all the default Keras RNN layers (LSTM/GRU), they all have a init param (go_backwards), and when it is configured to True, it will process the inputs in the reverse order.

You will need to update your custom RNN layer to have that init param as well if you want to use it with Bidirectional wrapper. So far the docstring for Bidirectional didn't state this point clearly, and we will update that.

As a workaround, you can also choose to implement a custom RNN cell, which define the math calculation for one single time step. Passing the cell to base RNN layer and wrap the RNN layer with Bidirectional wrapper, and the default RNN layer will handle the go_backwards correctly.

Jbiloki

comment created time in 14 days

Pull request review commenttensorflow/tensorflow

[ROCm] Support of GRU and LSTM

 def lstm_with_backend_selection(inputs, init_h, init_c, kernel,     sequence_lengths: The lengths of all sequences coming from a variable length       input, such as ragged tensors. If the input has a fixed timestep size,       this should be None.-    zero_output_for_mask: Boolean, whether to output zero for masked timestep.+    +    put_for_mask: Boolean, whether to output zero for masked timestep.

Why is this docstring being changed.

ekuznetsov139

comment created time in 14 days

Pull request review commenttensorflow/tensorflow

[ROCm] Support of GRU and LSTM

 def lstm_with_backend_selection(inputs, init_h, init_c, kernel,   }    def gpu_lstm_with_fallback(inputs, init_h, init_c, kernel, recurrent_kernel,-                             bias, mask, time_major, go_backwards, activation,-                             recurrent_activation, sequence_lengths):+                               bias, mask, time_major, go_backwards, activation,

Also, the indent seems to be incorrect.

ekuznetsov139

comment created time in 14 days

Pull request review commenttensorflow/tensorflow

[ROCm] Support of GRU and LSTM

 def gru_with_backend_selection(inputs, init_h, kernel, recurrent_kernel, bias,   }    def gpu_gru_with_fallback(inputs, init_h, kernel, recurrent_kernel, bias,-                            mask, time_major, go_backwards, activation,-                            recurrent_activation, sequence_lengths):+                              mask, time_major, go_backwards, activation,

The indent here seems to be incorrect.

ekuznetsov139

comment created time in 14 days

push eventqlzh727/community

qlzh727

commit sha 3cce978931330e9b01347d4f970c66e670fd148c

Update the RFC nubmer

view details

push time in 15 days

PR opened tensorflow/community

Reviewers
RFC: Standalone Keras Repository

Comment period is open till 2/25/2020.

Standalone Keras Repository

Status Proposed
Author(s) Qianli Zhu (scottzhu@google.com), Francois Chollet (fchollet@google.com)
Sponsor Karmel Allison (karmel@google.com)
Updated 2020-02-05

Objective

Move the Keras code from the TensorFlow main GitHub repository to its own repository, with TensorFlow as a dependency.

+351 -0

0 comment

1 changed file

pr created time in 15 days

push eventqlzh727/community

François Chollet

commit sha 0405d803912771e21eec0590cb70fb9d63fdc421

Update 20200205-standalone-keras-repository.md

view details

Qianli Scott Zhu

commit sha 706479408c695c14a7142186e37362fc958d6268

Merge pull request #2 from fchollet/patch-1 Proofreading, etc.

view details

push time in 15 days

PR merged qlzh727/community

Proofreading, etc.

Only small nits :)

+160 -132

1 comment

1 changed file

fchollet

pr closed time in 15 days

pull request commentqlzh727/community

Proofreading, etc.

Thanks for the quick review and update. Merging now.

fchollet

comment created time in 15 days

push eventqlzh727/community

qlzh727

commit sha 0145eea24e3a7d37d411caae1aba11f074d95875

Update wording and nits.

view details

push time in 15 days

push eventqlzh727/community

qlzh727

commit sha 7af3f0b67f091ed7108944cd0255cd57543d5995

Initial commit for RFC standalone Keras repository

view details

push time in 15 days

push eventqlzh727/community

Hanna Revinskaya

commit sha 2a4df4dbafd42722b5eab59475b542bdde19f351

RFC for TensorFlow API names changes

view details

Hanna Revinskaya

commit sha 0601bc9c03f6aee01197b65a417f2e153bd4c00f

Attempting to fix tables. Also removing Appendix links, they don't seem to work

view details

Hanna Revinskaya

commit sha 3f7d935a0f8822bd1f9b5a618b1f08e2d131217c

Removing namespace changes for symbols that we plan to deprecate. Also, mentioning that we want to move TensorFlow debugger to tf.debugging.

view details

Hanna Revinskaya

commit sha 34cc1014741479b4a3e8e8f76eb0e5e4cc0bf3c2

Remove tf.make_ndarray and tf.make_tensor_proto from proposed changes.

view details

Hanna Revinskaya

commit sha 1efdfe40a138ad072bf8fbe81e97201559866d5b

Keep tf.distributions.* symbols in their current namespace (i.e. don't move to tf.probability.*

view details

Rohan Jain

commit sha 9d5c52f7bb742bdb987cbbd6b94a781993f3755e

Adding a doc to deprecate collections

view details

Martin Wicke

commit sha f8dad65f3e536707add510a939fae2b47a1bc6e0

Add contrib deprecation RFC

view details

Martin Wicke

commit sha 879d13fcfe536371fdcbcc76ad78405ea731f3fa

move contrib/integrate to tensorflow/scientific

view details

Hanna Revinskaya

commit sha 768b828d486ab2227e1cd7f469e0b70a34bac5a2

Update document to indicate we plan to add tf.keras.losses, tf.keras.metrics, tf.keras.layers and deprecated symbols under tf.layers and tf.logging. I also removed tf.log_poisson_loss changes according to the comment as well as tf.global_norm - still need to decide where it needs to go.

view details

Hanna Revinskaya

commit sha b10e02782fd693b907673ef2dd3771bb2cd029e8

Replace tf.global_norm with tf.linalg.global_norm. Add endpoint for tf.bitcast in tf.dtypes.bitcast to match tf.cast changes

view details

Hanna Revinskaya

commit sha 25ab399ecf66f7cee8e7f8c479aefcb96f8cc96b

Keep aliases for tf.keras.layers in tf.layers. Also, add a note saying that signatures of tf.layers, tf.losses and tf.metrics will likely change when we add additional endpoints under tf.keras.layers to match existing functions under tf.keras better

view details

Martin Wicke

commit sha 834d6c2e7f03774c874f4ec3239da28cacf2f858

Added ignite and igfs

view details

Anna Revinskaya

commit sha 560f860f5278c4e7ecfd3c756ce2e76f43db7259

Deprecate tf.manip. Also, deprecate tf.sparse_to_dense. It is very similar to tf.sparse_tensor_to_dense except the latter takes SparseTensor as argument.

view details

Martin Wicke

commit sha d615e80262bd1fa49edc3c53029adeeaa9137f5d

Add @terrytangyuan to timeseries

view details

Martin Wicke

commit sha 5c805774c8b45a3a0ce1337ada4a3f1fc3307983

Update 20180907-contrib-sunset.md

view details

Saurabh Saxena

commit sha 296ed6e66674c6937ead279d265dfa244202feb6

RFC: Functional while_loop (#13) * RFC for differentiable functional while. * Formatting fixes * Formatting fixes * Fix link for if op lowering * Fix formatting of computing gradients section * gradients_impl.gradients -> tf.gradients * Formatting changes * skye's suggestions * Remove grappler pass for pruning. * fix typo * fix formatting * Add link for model_pruner * Clarify dangling while op point. * Link functional cond doc * Update last updated. * Make proposal as accepted and add discussion notes. * Add direct link to comment with notes. * Update Author(s) -> Author

view details

Edd Wilder-James

commit sha ce1b08dd52110b34623c891efe6c8a855431bb6f

RFC: Generalizing tf.data batching using windowing and reducers (#5) * Create 20180726-tf-data-windowing-reducers.md Proposed RFC from @jsimsa, addresses the known limitations of the current tf.data batching API: * it provides a mechanism for padded batching of sparse tensors * it facilitates customization of batching logic (users can now express batching logic as a pure Python function) * it enables application of different batching logic on different components * Update 20180726-tf-data-windowing-reducers.md Updates following the review committee. * Update RFC metadata Status => accepted. Update revision date.

view details

Jiri Simsa

commit sha de0ecb98c93c342bb1081329094be1fbe07030db

Minor fixes. (#19)

view details

Alexandre Passos

commit sha 8ad0109177ddd52d4de81d0ba4f861a688b421fa

RFC: Variables in TensorFlow 2.0 (#11) * changes * changes * fix link * checkpoint link

view details

Edd Wilder-James

commit sha 0e9b09ead67433bd55f78da164a47f2ef1653b92

Update Variables in 2.0 to Accepted

view details

push time in 15 days

PR merged qlzh727/community

Rebase to the HEAD of tf/community.
+22325 -59

0 comment

139 changed files

qlzh727

pr closed time in 15 days

PR opened qlzh727/community

Rebase to the HEAD of tf/community.
+22325 -59

0 comment

139 changed files

pr created time in 15 days

push eventtensorflow/addons

Ulf Hamster

commit sha cdb43ffd7e4b89d6ce8cdadcd62fec46c7f0f7fa

LayernormSimpleRNN moved to addons (#841) * LayernormSimpleRNN moved to addons * code-format run * use super instead of calling the parent class * deactivate layernorm's bias term (beta) for centering, and apply the normal self.bias term after scaling with layernorm for centering. docstring with explanatory formulas added to cell's call method * use_layernorm=True set as default * import alligned with cell.py, examples in docstring corrected * import aligned with cell_test.py * code for LayernormSimpleRNN moved into cell.py and cell_test.py * pylint errors corrected * bazel's timeout increased from small to large for cell_test.py * test with training deactivated * non-ascii char replaced * dict syntax for python2 changed * Renamed to LayerNorm... * direct parent class call replaced with super * error due to import change corrected * uncomment line * unit test added * Name change in unit test file * Still the class name change * deleted dtype and trainable args for parent class * remove self for super parent class calls * compare arrays with assertAllEqual * use_layernorm removed * dict removed from return statement * LayerNormSimpleRNN removed, use kwargs, comments removed * forward **kwargs to other layers * a more pythonic dict loop

view details

push time in 16 days

PR merged tensorflow/addons

LayernormSimpleRNN moved to addons cla: yes

Code moved from this PR https://github.com/tensorflow/tensorflow/pull/35469

+291 -0

3 comments

3 changed files

ulf1

pr closed time in 16 days

Pull request review commenttensorflow/addons

LayernormSimpleRNN moved to addons

 def _create_norm_layer(self, name):             gamma_initializer=self.norm_gamma_initializer,             epsilon=self.norm_epsilon,             name=name)+++@tf.keras.utils.register_keras_serializable(package='Addons')+class LayerNormSimpleRNNCell(keras.layers.SimpleRNNCell):+    """Cell class for LayerNormSimpleRNN.++    References:+    [1] Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton.+        "Layer Normalization." ArXiv:1607.06450 [Cs, Stat],+        July 21, 2016. http://arxiv.org/abs/1607.06450++    Arguments:+      units: Positive integer, dimensionality of the output space.+      activation: Activation function to use.+        Default: hyperbolic tangent (`tanh`).+        If you pass `None`, no activation is applied+        (ie. "linear" activation: `a(x) = x`).+      use_bias: Boolean, (default `True`), whether the layer uses a bias+        vector.+      layernorm_epsilon: Float, (default `1e-5`), Small float added to variance+        to avoid dividing by zero.+      kernel_initializer: Initializer for the `kernel` weights matrix,+        used for the linear transformation of the inputs. Default:+        `glorot_uniform`.+      recurrent_initializer: Initializer for the `recurrent_kernel`+        weights matrix, used for the linear transformation of the recurrent+        state. Default: `orthogonal`.+      bias_initializer: Initializer for the bias vector (`use_bias=True`).+         Default: `zeros`.+      gamma_initializer: Initializer for the gamma vector of the layer+         normalization layer. Default: `ones`.+      kernel_regularizer: Regularizer function applied to the `kernel` weights+        matrix. Default: `None`.+      recurrent_regularizer: Regularizer function applied to the+        `recurrent_kernel` weights matrix. Default: `None`.+      bias_regularizer: Regularizer function applied to the bias vector+         (`use_bias=True`). Default: `None`.+      gamma_regularizer: Regularizer function applied to the gamma vector+         of the layer normalization layer. Default: `None`.+      kernel_constraint: Constraint function applied to the `kernel` weights+        matrix. Default: `None`.+      recurrent_constraint: Constraint function applied to the+        `recurrent_kernel` weights matrix. Default: `None`.+      bias_constraint: Constraint function applied to the bias vector+         (`use_bias=True`). Default: `None`.+      gamma_constraint: Constraint function applied to the gamma vector+         of the layer normalization layer. Default: `None`.+      dropout: Float between 0 and 1. Fraction of the units to drop for the+        linear transformation of the inputs. Default: 0.+      recurrent_dropout: Float between 0 and 1. Fraction of the units to drop+        for the linear transformation of the recurrent state. Default: 0.++    Call arguments:+      inputs: A 2D tensor, with shape of `[batch, feature]`.+      states: A 2D tensor with shape of `[batch, units]`, which is the state+        from the previous time step. For timestep 0, the initial state provided+        by the user will be feed to cell.+      training: Python boolean indicating whether the layer should behave in+        training mode or in inference mode. Only relevant when `dropout` or+        `recurrent_dropout` is used.++    Examples:++    ```python+    import numpy as np+    import tensorflow.keras as keras+    import tensorflow_addons as tfa++    inputs = np.random.random([32, 10, 8]).astype(np.float32)+    rnn = keras.layers.RNN(tfa.rnn.LayerNormSimpleRNNCell(4))++    output = rnn(inputs)  # The output has shape `[32, 4]`.++    rnn = keras.layers.RNN(+        tfa.rnn.LayerNormSimpleRNNCell(4),+        return_sequences=True,+        return_state=True)++    # whole_sequence_output has shape `[32, 10, 4]`.+    # final_state has shape `[32, 4]`.+    whole_sequence_output, final_state = rnn(inputs)+    ```+    """++    def __init__(+            self,+            units,+            activation='tanh',+            use_bias=True,+            layernorm_epsilon=1e-05,  # NEW(!)+            kernel_initializer='glorot_uniform',+            recurrent_initializer='orthogonal',+            bias_initializer='zeros',+            gamma_initializer='ones',  # NEW(!)+            kernel_regularizer=None,+            recurrent_regularizer=None,+            bias_regularizer=None,+            gamma_regularizer=None,  # NEW(!)+            kernel_constraint=None,+            recurrent_constraint=None,+            bias_constraint=None,+            gamma_constraint=None,  # NEW(!)+            dropout=0.,+            recurrent_dropout=0.,+            **kwargs):+        super(LayerNormSimpleRNNCell, self).__init__(+            units,+            activation=activation,+            use_bias=use_bias,+            kernel_initializer=kernel_initializer,+            recurrent_initializer=recurrent_initializer,+            bias_initializer=bias_initializer,+            kernel_regularizer=kernel_regularizer,+            recurrent_regularizer=recurrent_regularizer,+            bias_regularizer=bias_regularizer,+            kernel_constraint=kernel_constraint,+            recurrent_constraint=recurrent_constraint,+            bias_constraint=bias_constraint,+            dropout=dropout,+            recurrent_dropout=recurrent_dropout,+            dtype=kwargs.get('dtype'),+            trainable=kwargs.get('trainable', True))+        self.layernorm = keras.layers.LayerNormalization(+            axis=-1,+            epsilon=layernorm_epsilon,+            center=False,+            scale=True,+            beta_initializer=None,

ok, I think let layernorm to use its default value should be fine.

ulf1

comment created time in 16 days

issue commenttensorflow/tensorflow

Potential memory leak when using LSTM + TimeDistributed

yes, the issue is resolved by d064c6f

algrmur

comment created time in 16 days

issue commenttensorflow/tensorflow

[TF 2.1] Error when converting LSTM model to a frozen graph using convert_variables_to_constants_v2()

Hi @jakesabathia2, the code snippet you have is using a lot of internal python function, which access a lot implementation details and we don't ensure any API guarantee for them.

Can you provide more context about why you need to access the frozen graph? Also please check the model saving doc here https://www.tensorflow.org/guide/saved_model.

jakesabathia2

comment created time in 18 days

issue closedtensorflow/tensorflow

Keras RNN training speed significantly slower with eager execution/control flow v2

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
  • TensorFlow installed from (source or binary): source
  • TensorFlow version (use command below): 2.0.0
  • Python version: 3.6.8
  • Bazel version (if compiling from source): N/A
  • GCC/Compiler version (if compiling from source): N/A
  • CUDA/cuDNN version: 10.0/7.6
  • GPU model and memory: GTX 980 Ti

Describe the current behavior

Enabling eager execution or control flow v2 causes RNN training speed to decrease significantly.

Describe the expected behavior

Enabling eager mode or control flow v2 should not affect the training time (or improve it, ideally).

Code to reproduce the issue

import tensorflow as tf
import numpy as np
import timeit

use_eager = False
use_v2 = False

if not use_eager:
    tf.compat.v1.disable_eager_execution()
if not use_v2:
    tf.compat.v1.disable_control_flow_v2()


n_steps = 1000
n_input = 100
n_hidden = 1000
batch_size = 64

inputs = tf.keras.Input((n_steps, n_input))
outputs = tf.keras.layers.SimpleRNN(units=n_hidden, return_sequences=True)(inputs)
outputs = tf.keras.layers.Dense(units=n_input)(outputs)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

model.compile(optimizer=tf.optimizers.SGD(0.1), loss="mse")

x = np.ones((batch_size, n_steps, n_input))
y = np.ones((batch_size, n_steps, n_input))

# warmup
model.fit(x, y, epochs=1)

start = timeit.default_timer()
model.fit(x, y, epochs=10)
print("Execution time:", timeit.default_timer() - start)

Other info / logs On my machine the results look like:

  • use_eager=False, use_v2=False: 5.90s
  • use_eager=False, use_v2=True: 8.08s
  • use_eager=True, use_v2=False: 9.81s
  • use_eager=True, use_v2=True: 10.10s

So, overall a >60% increase in training time comparing no eager and no v2 to the current defaults.

closed time in 18 days

drasmuss

issue commenttensorflow/tensorflow

Keras RNN training speed significantly slower with eager execution/control flow v2

With all the recent update for performance, the v1/v2 and eager/graph training time is almost identical to each other.

Please see the results in https://colab.research.google.com/gist/qlzh727/5bf1e16285bfcf2b3c2aea506fffd2e8/untitled188.ipynb

I update the original test script to fit on 10 batches instead of 1 batch in each epoch, which should simulate more standard use case.

drasmuss

comment created time in 18 days

issue commenttensorflow/tensorflow

Bidirectional LSTM fail on TF2.0

I think this might be a very specific cudnn kernel issue. Currently I can't reproduce it with TF nightly on a GPU colab. For any of you still facing the issue, could you try the TF nightly version as well?

PierrePivert

comment created time in 18 days

issue closedtensorflow/tensorflow

Tensorflow 2.1 Tracing Error with LSTM layer

I am implementing a tensorflow model using the keras LSTM layer.

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu
  • TensorFlow version (use command below): 2.1.0
  • Python version: 3.7
  • GPU model and memory: CPU

Describe the current behavior Method Model.predict raise warning about Tracing WARNING:tensorflow:5 out of the last 5 calls to <function _make_execution_function.<locals>.distributed_function at 0x7eff0c74def0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

Describe the expected behavior Same as Model.predict_on_batch

Code to reproduce the issue

import tensorflow as tf
import numpy as np
import time

feature_dimension=2
lstm_size=256
learning_rate=0.0001

model = tf.keras.Sequential([
    tf.keras.Input([None, feature_dimension]),
    tf.keras.layers.LSTM(lstm_size),
    tf.keras.layers.Dense(1, activation='linear')
])

model.compile(loss='mean_squared_error',
              optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),
              metrics=['mse', 'mae'])

print('Model.predict')
st = time.time()
for i in range(5,100):
    model.predict(np.array([[(0.1,0.1)]*i]*10))
print(f'Model.predict in {time.time() - st}')

print('Model.predict_on_batch')
st = time.time()
for i in range(5,100):
    np.array(model.predict_on_batch(np.array([[(0.1,0.1)]*i]*10)))

print(f'Model.predict_on_batch in {time.time() - st}')

Other info / logs For me help use predict_on_batch method.

closed time in 21 days

michal-bouska

issue commenttensorflow/tensorflow

Tensorflow 2.1 Tracing Error with LSTM layer

I don't think this is the issue with LSTM. The warning also shows up when LSTM layer is removed.

The real issue is that the usage between predict() and predict_on_batch(). Since the predict() is designed to handle large data and process them one batch by one batch, under the hood, we build TF functions. Whereas for predict_on_batch, since there is only one batch, we use more light weighted approach.

For this warning, it is expected since your input shape changes everytime, and model as to retrace the function based on that change.

See https://www.tensorflow.org/api_docs/python/tf/keras/Model?version=nightly#predict for more details and performance implication of small and large data input.

michal-bouska

comment created time in 21 days

issue commenttensorflow/tensorflow

Recurrent Dropout is Wrong

@OverLordGoldDragon, I would like to have @fchollet to confirm this is ok to proceed as the API owner.

OverLordGoldDragon

comment created time in 21 days

more