profile
viewpoint

keras-team/keras 46899

Deep Learning for humans

Vishal-V/GSoC 5

Google Summer of Code 2019 with Tensorflow: Final Work Product

k-w-w/community 0

Stores documents used by the TensorFlow developer community

k-w-w/models 0

Models and examples built with TensorFlow

k-w-w/reference 0

Reference implementations of MLPerf benchmarks

k-w-w/tensor2tensor 0

A library for generalized sequence to sequence models

k-w-w/tensorboard 0

TensorFlow's Visualization Toolkit

k-w-w/tensorflow 0

An Open Source Machine Learning Framework for Everyone

MehdiShz/reference 0

Reference implementations of MLPerf benchmarks

issue commenttensorflow/tensorflow

Re-compiling keras model causes save / load_model to fail

I believe this issue should be fixed soon, I have a change waiting to be reviewed.

ding-c3

comment created time in 2 days

Pull request review commenttensorflow/tensorflow

Update save.py

 from __future__ import division from __future__ import print_function -import os import sys  import six -from tensorflow.python import tf2 from tensorflow.python.keras.saving import hdf5_format from tensorflow.python.keras.saving.saved_model import load as saved_model_load from tensorflow.python.keras.saving.saved_model import save as saved_model_save from tensorflow.python.saved_model import loader_impl+from tensorflow.python.util.lazy_loader import LazyLoader from tensorflow.python.util.tf_export import keras_export +network = LazyLoader(+    'network', globals(),+    'tensorflow.python.keras.engine.network')

There's an internal bug, so this has to use double quotes:

# pylint:disable=g-inconsistent-quotes
network = LazyLoader(
    "network", globals(),
    "tensorflow.python.keras.engine.network")
# pylint:enable=g-inconsistent-quotes
nikochiko

comment created time in 2 days

issue commenttensorflow/tensorflow

[TF 2.1] Error when converting LSTM model to a frozen graph using convert_variables_to_constants_v2()

+@gargn Should convert_variables_to_constants_v2 be working for this case?

jakesabathia2

comment created time in 7 days

issue commenttensorflow/tensorflow

Custom layer weights all have the same name by default

Thanks for reporting this! This is one of the effects of moving into TF 2.X -- variable names are no longer uniqueified. This should be fixed in one of two places: either when the variables are created, or when the variables are saved out to HDF5.

(This should only effect the HDF5 saving format. The tf format should still work)

zaccharieramzi

comment created time in 9 days

issue commenttensorflow/tensorflow

Exporting custom models with SavedModel format

When dynamic is set to True, it means the model is only able to run in eager mode (and thus incompatible with graphs). The SavedModel format tries to serialize the model in a graph, so this is fundamentally incompatible.

The error being raised here is actually related to generator.predict(test_x). (try calling this right after the generator and test_x are created). I've attached the error logs below. @qlzh727 Can you take a look and see if it's an error with the _aggregate_predict_results? If there's no issue, how can the error be improved?

<ipython-input-4-4f5b3ec7f8fa> in export_model()
    265     test_x = tf.random.normal([1, noise_dim])
    266     _ = generator(test_x, training=False)
--> 267     _ = generator.predict(test_x)
    268     checkpoint_dir = './models/dcgan'
    269     checkpoint = tf.train.Checkpoint(generator=generator)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in predict(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)
   1011         max_queue_size=max_queue_size,
   1012         workers=workers,
-> 1013         use_multiprocessing=use_multiprocessing)
   1014 
   1015   def reset_metrics(self):

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py in predict(self, model, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing, **kwargs)
    496         model, ModeKeys.PREDICT, x=x, batch_size=batch_size, verbose=verbose,
    497         steps=steps, callbacks=callbacks, max_queue_size=max_queue_size,
--> 498         workers=workers, use_multiprocessing=use_multiprocessing, **kwargs)
    499 
    500 

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py in _model_iteration(self, model, mode, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, **kwargs)
    473               mode=mode,
    474               training_context=training_context,
--> 475               total_epochs=1)
    476           cbks.make_logs(model, epoch_logs, result, mode)
    477 

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
    166       else:
    167         batch_outs = training_v2_utils._aggregate_predict_results(
--> 168             strategy, batch_outs, model)
    169 
    170       if step == 0:

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in _aggregate_predict_results(strategy, batch_outs, model)
    262     nested_outs = batch_outs[i * num_replicas:i * num_replicas + num_replicas]
    263     per_output_result = dist_utils.concat_along_batch_dimension(
--> 264         nest.flatten(nested_outs))
    265 
    266     if need_batch_index_gather:

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py in concat_along_batch_dimension(outputs)
   1197 def concat_along_batch_dimension(outputs):
   1198   """Concats prediction outputs along the batch dimension."""
-> 1199   if isinstance(outputs[0], sparse_tensor.SparseTensor):
   1200     return sparse_ops.sparse_concat_v2(axis=0, sp_inputs=outputs)
   1201   if isinstance(outputs[0], ragged_tensor.RaggedTensor):

IndexError: list index out of range
moono

comment created time in 16 days

PR closed tensorflow/tensorflow

Reviewers
fix(keras): load_model should pass custom_objects when loading models in tf format cla: yes comp:keras size:S
# this fails if the loaded model is a SavedModel (saved with format="tf") 
from tensorflow.keras import models
model = models.load_model("/path/to/tf_model", custom_objects={"custom_loss": custom_loss})

The custom_object should be passed to the compile correctly (as it is with when loading a h5 file).

Should fix: #33646

+9 -3

5 comments

2 changed files

j-o-d-o

pr closed time in 17 days

pull request commenttensorflow/tensorflow

fix(keras): load_model should pass custom_objects when loading models in tf format

Thanks for the PR! A similar fix was submitted and this issue should be resolved in nightly

j-o-d-o

comment created time in 17 days

issue closedtensorflow/tensorflow

Loaded model fails on inference

System information

  • TF version 2.1.0, and I have also has this error on TF 2.2.0-dev20200114
  • Python 3.7

Describe the current behavior

  • Similar to #35527, when I save and then load my model, it fails upon actually using the model, citing the inputs being different to what was expected.
  • When I run my code (see below) I get the following error:
ValueError: Could not find matching function to call loaded from the SavedModel. Got:
  Positional arguments (3 total):
    * (<tf.Tensor 'inputs:0' shape=(1, 10) dtype=int32>, <tf.Tensor 'inputs_1:0' shape=(1, 10) dtype=int32>)
    * False
    * None
  Keyword arguments: {}

Expected these arguments to match one of the following 4 option(s):

Option 1:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='input_ids'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='attention_mask')]
    * True
    * None
  Keyword arguments: {}

Option 2:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='input_ids'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='attention_mask')]
    * False
    * None
  Keyword arguments: {}

Option 3:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1')]
    * True
    * None
  Keyword arguments: {}

Option 4:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1')]
    * False
    * None
  Keyword arguments: {}
  • My reasoning for making this issue when #35527 already exists is that my code to reproduce the issue is more succinct and so should hopefully be easier to troubleshoot.

Describe the expected behavior

  • The model should load and behave in the exact same manner in which it was saved (I.e. not crash when doing inference on data). This is currently not the case.

Code to reproduce the issue



############# Create a model using TF and the popular transformers NLP package ###########

class TagModelCreator:

    def __init__(self, language_model):
        self.language_model = language_model

    def create(self, num_classes, max_seq_len, get_token_type_ids=False):

        input_modules = []
        
        input_modules.append(tf.keras.layers.Input(shape=(max_seq_len), dtype='int32', name='input_ids'))
        input_modules.append(tf.keras.layers.Input(shape=(max_seq_len), dtype="int32", name='attention_mask'))

        lang_layer = self.language_model(input_modules)
        linear_layer = tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(num_classes, name='classifier'))(lang_layer[0])
        model = tf.keras.Model(inputs=input_modules, outputs=linear_layer)

        return model

from transformers import TFAutoModel

model_name = "bert-base-uncased"
language_model = TFAutoModel.from_pretrained(model_name)
tagging_model_creator = TagModelCreator(language_model)
arbitrary_class_num = 2
arbitrary_sequence_length = 10
tagging_model = tagging_model_creator.create(arbitrary_class_num, arbitrary_sequence_length)




######### Create some spoof data to see how the model handles the data ####################

def data_generator():
    yield (([0]*arbitrary_sequence_length, [1]*arbitrary_sequence_length))

input_types = ((tf.int32, tf.int32))
input_shape = ((tf.TensorShape([None]), tf.TensorShape([None])))

tf_dataset = tf.data.Dataset.from_generator(data_generator, input_types, input_shape).batch(7)






######### Use the spoof data on the model, to confirm that it does inference on the data without errors########

for example_input in tf_dataset:
    test_output = tagging_model(example_input)
    break

print(test_output)
print("Inference is done correctly BEFORE re-loading the model")






######## Save and reload the model #################

tf.keras.models.save_model(model=tagging_model,
                                       filepath="test_model_save.tf",
                                       save_format="tf",
                                       include_optimizer=True
                                      )

reloaded_model = tf.keras.models.load_model(filepath="test_model_save.tf")






####### Try to repeat the inference as above #########
for example_input in tf_dataset:
    test_output = reloaded_model(example_input)
    break

Other info / logs The full output of the above code, including printouts and the error stack trace is

tf.Tensor(
[[[-0.6191008  -0.12756673]
  [-0.89110005  0.06499487]
  [-0.8666591  -0.02111167]
  [-0.8456675  -0.08551306]
  [-0.853022   -0.15643758]
  [-0.8632274  -0.20486367]
  [-0.8571876  -0.24682882]
  [-0.8400811  -0.2774819 ]
  [-0.8864943  -0.32766515]
  [-0.8612056  -0.3529073 ]]], shape=(1, 10, 2), dtype=float32)
Inference is done correctly BEFORE re-loading the model
WARNING:tensorflow:From C:\Users\Peter\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1786: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
INFO:tensorflow:Assets written to: test_model_save.tf\assets
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-9d120ebc3322> in <module>
     80 ####### Try to repeat the inference as above #########
     81 for example_input in tf_dataset:
---> 82     test_output = reloaded_model(example_input)
     83     break
     84 

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\keras\engine\base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\keras\saving\saved_model\utils.py in return_outputs_and_add_losses(*args, **kwargs)
     57     inputs = args[inputs_arg_index]
     58     args = args[inputs_arg_index + 1:]
---> 59     outputs, losses = fn(inputs, *args, **kwargs)
     60     layer.add_loss(losses, inputs)
     61     return outputs

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\keras\saving\saved_model\utils.py in wrap_with_training_arg(*args, **kwargs)
    111         training,
    112         lambda: replace_training_and_call(True),
--> 113         lambda: replace_training_and_call(False))
    114 
    115   # Create arg spec for decorated function. If 'training' is not defined in the

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\keras\utils\tf_utils.py in smart_cond(pred, true_fn, false_fn, name)
     57         pred, true_fn=true_fn, false_fn=false_fn, name=name)
     58   return smart_module.smart_cond(
---> 59       pred, true_fn=true_fn, false_fn=false_fn, name=name)
     60 
     61 

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\smart_cond.py in smart_cond(pred, true_fn, false_fn, name)
     54       return true_fn()
     55     else:
---> 56       return false_fn()
     57   else:
     58     return control_flow_ops.cond(pred, true_fn=true_fn, false_fn=false_fn,

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\keras\saving\saved_model\utils.py in <lambda>()
    111         training,
    112         lambda: replace_training_and_call(True),
--> 113         lambda: replace_training_and_call(False))
    114 
    115   # Create arg spec for decorated function. If 'training' is not defined in the

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\keras\saving\saved_model\utils.py in replace_training_and_call(training)
    106     def replace_training_and_call(training):
    107       set_training_arg(training, training_arg_index, args, kwargs)
--> 108       return wrapped_call(*args, **kwargs)
    109 
    110     return tf_utils.smart_cond(

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\eager\def_function.py in __call__(self, *args, **kwds)
    566         xla_context.Exit()
    567     else:
--> 568       result = self._call(*args, **kwds)
    569 
    570     if tracing_count == self._get_tracing_count():

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\eager\def_function.py in _call(self, *args, **kwds)
    604       # In this case we have not created variables on the first call. So we can
    605       # run the first trace but we should fail if variables are created.
--> 606       results = self._stateful_fn(*args, **kwds)
    607       if self._created_variables:
    608         raise ValueError("Creating variables on a non-first call to a function"

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\eager\function.py in __call__(self, *args, **kwargs)
   2360     """Calls a graph function specialized to the inputs."""
   2361     with self._lock:
-> 2362       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
   2363     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
   2364 

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\eager\function.py in _maybe_define_function(self, args, kwargs)
   2701 
   2702       self._function_cache.missed.add(call_context_key)
-> 2703       graph_function = self._create_graph_function(args, kwargs)
   2704       self._function_cache.primary[cache_key] = graph_function
   2705       return graph_function, args, kwargs

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\eager\function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2591             arg_names=arg_names,
   2592             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2593             capture_by_value=self._capture_by_value),
   2594         self._function_attributes,
   2595         # Tell the ConcreteFunction to clean up its graph once it goes out of

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    976                                           converted_func)
    977 
--> 978       func_outputs = python_func(*func_args, **func_kwargs)
    979 
    980       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\eager\def_function.py in wrapped_fn(*args, **kwds)
    437         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    438         # the function a weak reference to itself to avoid a reference cycle.
--> 439         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    440     weak_wrapped_fn = weakref.ref(wrapped_fn)
    441 

~\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\saved_model\function_deserialization.py in restored_function_body(*args, **kwargs)
    260         .format(_pretty_format_positional(args), kwargs,
    261                 len(saved_function.concrete_functions),
--> 262                 "\n\n".join(signature_descriptions)))
    263 
    264   concrete_function_objects = []

ValueError: Could not find matching function to call loaded from the SavedModel. Got:
  Positional arguments (3 total):
    * (<tf.Tensor 'inputs:0' shape=(1, 10) dtype=int32>, <tf.Tensor 'inputs_1:0' shape=(1, 10) dtype=int32>)
    * False
    * None
  Keyword arguments: {}

Expected these arguments to match one of the following 4 option(s):

Option 1:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='input_ids'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='attention_mask')]
    * True
    * None
  Keyword arguments: {}

Option 2:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='input_ids'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='attention_mask')]
    * False
    * None
  Keyword arguments: {}

Option 3:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1')]
    * True
    * None
  Keyword arguments: {}

Option 4:
  Positional arguments (3 total):
    * [TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1')]
    * False
    * None
  Keyword arguments: {}

closed time in 17 days

Peter-Devine

issue commenttensorflow/tensorflow

Loaded model fails on inference

Thanks for reporting this bug, this should be fixed in the latest nightly.

(also, you can you triple ``` to format multiple lines of code)

Peter-Devine

comment created time in 17 days

issue closedtensorflow/tensorflow

load_weights h5file failed (cause by BN layer) in tensorflow2.0

<em>Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em>

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): win10
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): v2.0.0-rc1-51-g2646d23074 2.0.0-rc2
  • Python version: 3.6
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: NO
  • GPU model and memory: NO

Describe the current behavior I save my model using .h5 format(model.save_weights("./save_weights/resNet.h5")). And I load this weights(model.load_weights('./save_weights/resNet.h5', by_name=True)) then failed. this error is shape mismatch. i did some tests: (1) when i save model using ckpt format, i can successfully load weights. (2) when i save model that no use bn layer using h5 format, i can successfully load weights.

Describe the expected behavior i think i should load model weights(include bn layer) using .h5 format.

Code to reproduce the issue (1) i tracked the problem. the problem is in "\tensorflow_core\python\keras\saving\hdf5_format.py" file(754 line, "if K.int_shape(symbolic_weights[i]) != weight_values[i].shape:"). (2) when i load model weights, i find the variables order(layers order) is different between "symbolic_weights" and "weight_values" . so you can't simply use list index to load weights. (3) the main reason is caused by bn layer. i check the varables "symbolic_weights" and "weight_values", the bn parameters order in "weight_values" is like [gamma, bata, mean, variance, gamma, bata, mean, variance] but in "symbolic_weights" is like [gamma, bata, gamma, bata, mean, variance, mean, variance].

Other info / logs Traceback (most recent call last): File "E:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\network.py", line 1175, in load_weights saving.load_weights_from_hdf5_group_by_name(f, self.layers) File "E:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\saving\hdf5_format.py", line 760, in load_weights_from_hdf5_group_by_name str(weight_values[i].shape) + '.') ValueError: Layer #4 (named "block1"), weight <tf.Variable 'block1/unit_1/conv2/kernel:0' shape=(3, 3, 64, 64) dtype=float32, numpy= array([[[[ 0.00050041, -0.06431282, -0.00577746, ..., -0.05891485, -0.02970275, -0.06618612], ....... -0.04930668, -0.02927044]]]], dtype=float32)> has shape (3, 3, 64, 64), but the saved weight has shape (64,).

closed time in 22 days

WZMIAOMIAO

issue commenttensorflow/tensorflow

load_weights h5file failed (cause by BN layer) in tensorflow2.0

Currently hdf5 weights are saved with a specific ordering that depends on whether layers/variables are trainable. It's difficult to change this behavior as it would break all currently existing checkpoints.

Can you set trainable to False after the weights have been loaded? Alternatively, stick to using the ckpt format. Unfortunately there's not much we can do to fix this for hdf5.

WZMIAOMIAO

comment created time in 22 days

issue closedtensorflow/tensorflow

model restore error/confict in tensorflow

<em>Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em>

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): sudo pip3 install tensorflow-gpu==2.0.0
  • Python version: 3.6
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: 10.0.130/7.6.2
  • GPU model and memory:

You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior I have saved one model (with more than two layers) using model.save_weights and want to restore its weights and only use the front conv layers. But when I using model.load_weights, I got two different results:

  1. First, when I load weights only for the front conv layers, one bigger model can load successfully but another smaller model can not load.
  2. Two, the bigger model load success but the weights of the last layer is wrong but any other layers weiight is right.

Describe the expected behavior

I want to load a .ckpt into a smaller model successfully.

Code to reproduce the issue Here is my bigger model definition:

import tensorflow as tf
import math

NUM_CLASSES = 10

def swish(x):
    return x * tf.keras.activations.sigmoid(x)

def round_filters(filters, multiplier):
    depth_divisor = 8
    min_depth = None
    min_depth = min_depth or depth_divisor
    filters = filters * multiplier
    new_filters = max(min_depth, int(filters + depth_divisor / 2) // depth_divisor * depth_divisor)
    if new_filters < 0.9 * filters:
        new_filters += depth_divisor
    return int(new_filters)

def round_repeats(repeats, multiplier):
    if not multiplier:
        return repeats
    return int(math.ceil(multiplier * repeats))

def SEBlock(inputs, input_channels, ratio=0.25):

    num_reduced_filters = max(1, int(input_channels * ratio))
    branch = tf.keras.layers.GlobalAveragePooling2D()(inputs)
    # branch = tf.keras.layers.Lambda(lambda branch: tf.expand_dims(input=branch, axis=1))(branch)
    branch = tf.keras.backend.expand_dims(branch, 1)
    branch = tf.keras.backend.expand_dims(branch, 1)
    # branch = tf.keras.layers.Lambda(lambda branch: tf.expand_dims(input=branch, axis=1))(branch)
    branch = tf.keras.layers.Conv2D(filters=num_reduced_filters, kernel_size=(1, 1), strides=1, padding="same")(branch)
    branch = swish(branch)
    branch = tf.keras.layers.Conv2D(filters=input_channels, kernel_size=(1, 1), strides=1, padding='same')(branch)
    branch = tf.keras.activations.sigmoid(branch)
    output = inputs * branch

    return output

def MBConv(in_channels, out_channels, expansion_factor, stride, k, drop_connect_rate, inputs, training=False):
    x = tf.keras.layers.Conv2D(filters=in_channels * expansion_factor,kernel_size=(1, 1),strides=1,padding="same")(inputs)
    x = tf.keras.layers.BatchNormalization()(x, training=training)
    x = swish(x)
    x = tf.keras.layers.DepthwiseConv2D(kernel_size=(k, k), strides=stride, padding="same")(x)
    x = tf.keras.layers.BatchNormalization()(x, training=training)
    x = SEBlock(x, in_channels*expansion_factor)
    x = swish(x)
    x = tf.keras.layers.Conv2D(filters=out_channels,kernel_size=(1, 1),strides=1,padding="same")(x)
    x = tf.keras.layers.BatchNormalization()(x, training=training)
    if stride == 1 and in_channels == out_channels:
        if drop_connect_rate:
            x = tf.keras.layers.Dropout(rate=drop_connect_rate)(x, training=training)
        x = tf.keras.layers.Add()([x, inputs])

    return x

def build_mbconv_block(inputs, in_channels, out_channels, layers, stride, expansion_factor, k, drop_connect_rate, training):

    x = inputs
    for i in range(layers):
        if i == 0:
            x = MBConv(in_channels=in_channels, out_channels=out_channels, expansion_factor=expansion_factor,
                       stride=stride, k=k, drop_connect_rate=drop_connect_rate, inputs=x, training=training)
        else:
            x = MBConv(in_channels=out_channels, out_channels=out_channels, expansion_factor=expansion_factor,
                       stride=1, k=k, drop_connect_rate=drop_connect_rate, inputs=x, training=training)

    return x


def EfficientNet(inputs, width_coefficient, depth_coefficient, dropout_rate, drop_connect_rate=0.2, training=False):

    features = []

    x = tf.keras.layers.Conv2D(filters=round_filters(32, width_coefficient),kernel_size=(3, 3),strides=2, padding="same") (inputs)
    x = tf.keras.layers.BatchNormalization()(x, training=training)
    x = swish(x)

    x = build_mbconv_block(x, in_channels=round_filters(32, width_coefficient),
                           out_channels=round_filters(16, width_coefficient),
                           layers=round_repeats(1, depth_coefficient),
                           stride=1,
                           expansion_factor=1, k=3,
                           drop_connect_rate=drop_connect_rate,
                           training=training)
    features.append(x)

    x = build_mbconv_block(x, in_channels=round_filters(16, width_coefficient),
                           out_channels=round_filters(24, width_coefficient),
                           layers=round_repeats(2, depth_coefficient),
                           stride=1,
                           expansion_factor=6, k=3,
                           drop_connect_rate=drop_connect_rate,
                           training=training)
    features.append(x)

    x = build_mbconv_block(x, in_channels=round_filters(24, width_coefficient),
                           out_channels=round_filters(40, width_coefficient),
                           layers=round_repeats(2, depth_coefficient),
                           stride=2,
                           expansion_factor=6, k=5,
                           drop_connect_rate=drop_connect_rate,
                           training=training)
    features.append(x)

    x = build_mbconv_block(x, in_channels=round_filters(40, width_coefficient),
                           out_channels=round_filters(80, width_coefficient),
                           layers=round_repeats(3, depth_coefficient),
                           stride=2,
                           expansion_factor=6, k=3,
                           drop_connect_rate=drop_connect_rate,
                           training=training)
    features.append(x)

    x = build_mbconv_block(x, in_channels=round_filters(80, width_coefficient),
                           out_channels=round_filters(112, width_coefficient),
                           layers=round_repeats(3, depth_coefficient),
                           stride=1,
                           expansion_factor=6, k=5,
                           drop_connect_rate=drop_connect_rate,
                           training=training)
    features.append(x)

    x = build_mbconv_block(x, in_channels=round_filters(112, width_coefficient),
                           out_channels=round_filters(192, width_coefficient),
                           layers=round_repeats(4, depth_coefficient),
                           stride=2,
                           expansion_factor=6, k=5,
                           drop_connect_rate=drop_connect_rate,
                           training=training)
    features.append(x)

    x = build_mbconv_block(x, in_channels=round_filters(192, width_coefficient),
                           out_channels=round_filters(320, width_coefficient),
                           layers=round_repeats(1, depth_coefficient),
                           stride=1,
                           expansion_factor=6, k=3,
                           drop_connect_rate=drop_connect_rate,
                           training=training)
    features.append(x)

    x = tf.keras.layers.Conv2D(filters=round_filters(1280, width_coefficient), kernel_size=(1, 1), strides=1, padding='same')(x)
    x = tf.keras.layers.BatchNormalization()(x, training=training)
    x = swish(x)
    x = tf.keras.layers.GlobalAveragePooling2D()(x)
    x = tf.keras.layers.Dropout(rate=dropout_rate)(x, training=training)
    x = tf.keras.layers.Dense(units=1, activation=tf.keras.activations.softmax)(x)

    return x, features


def efficient_net_b0(inputs, training):
    return EfficientNet(inputs,
                        width_coefficient=1.0,
                        depth_coefficient=1.0,
                        dropout_rate=0.2,
                        drop_connect_rate=0.2,
                        training=training)

def up_sample(inputs, training=True):
    x = tf.keras.layers.UpSampling2D()(inputs)
    x = tf.keras.layers.BatchNormalization()(x, training=training)
    x = tf.keras.layers.ReLU()(x)
    return x

def biggerModel(inputs, outc, training=True):

    _, features =  efficient_net_b0(inputs=inputs, training=training)

    # [ 1/2, 1/4, 1/8, 1/8, 1/16]
    outputs = []
    for i, name in enumerate(features):
        x = features[i]
        if x.shape[1] > inputs.shape[1] // 4:
            continue
        while x.shape[1] < (inputs.shape[1]//4):
            x = up_sample(x, training)
        outputs.append(x)

    quater_res = tf.keras.layers.Concatenate()(outputs)
    quater_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(quater_res)
    quater_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(quater_res)
    quater_res_out = tf.keras.layers.Conv2D(outc, 1, 1, 'same', name='quater', activation=None)(quater_res)

    half_res = up_sample(quater_res, training)
    half_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(half_res)
    half_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(half_res)
    half_res_out = tf.keras.layers.Conv2D(outc, 1, 1, 'same', name='half', activation=None)(half_res)
    
    if training:
        return quater_res_out, half_res_out
    else:
        return quater_res_out

And here is my smaller model definition:

def smallModel(inputs, outc, training=True):


    quater_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(inputs)
    quater_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(quater_res)
    quater_res_out = tf.keras.layers.Conv2D(outc, 1, 1, 'same',  activation=None)(quater_res)


    half_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(quater_res)
    half_res = tf.keras.layers.Conv2D(512, 3, 1, 'same', activation=tf.nn.relu)(half_res)
    half_res_out = tf.keras.layers.Conv2D(outc, 1, 1, 'same', activation=None)(half_res)

    if training:
        return quater_res_out, half_res_out
    else:
        return quater_res_out

About bigger model, first, I save weights and print layer named 'quater' weights :

if __name__ == '__main__':
    import os
    os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
    inputs = tf.keras.Input(shape=(224, 224, 3), name='modelInput')
    outputs = biggerModel(inputs, outc=18 + 1, training=True)
    model = tf.keras.Model(inputs, outputs)
    # model.summary()

    model.save_weights('models/test/test')
    # model.load_weights('models/test/test')

    print(model.get_layer('quater').get_weights()[0][0][0][0:4])

and I get this: image and then I restore weight by setting parameter training = False and print quater layer weights:

if __name__ == '__main__':

    import os
    os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
    inputs = tf.keras.Input(shape=(224, 224, 3), name='modelInput')
    outputs = biggerModel(inputs, outc=18 + 1, training=False)
    model = tf.keras.Model(inputs, outputs)
    # model.summary()

    # model.save_weights('models/test/test')
    model.load_weights('models/test/test')

    print(model.get_layer('quater').get_weights()[0][0][0][0:4])

and got result like this: image We can see that this two outputs are different, but I have checkout all other layers weights are same resutlts, so, it's really weird.

For smaller model, as like in bigger model I have done, first save weights and print quater layer weights:


if __name__ == '__main__':

    import os
    os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
    inputs = tf.keras.Input(shape=(224, 224, 3), name='modelInput')
    outputs = smallModel(inputs, outc=18 + 1, training=True)
    model = tf.keras.Model(inputs, outputs)
    # model.summary()

    # model.save_weights('models/test/test')
    model.load_weights('models/test/test')

    print(model.get_layer('quater').get_weights()[0][0][0][0:4])

and result is: image Then I tried to restore this and setting parameter training = False:

if __name__ == '__main__':

    import os
    os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
    inputs = tf.keras.Input(shape=(224, 224, 3), name='modelInput')
    outputs = smallModel(inputs, outc=18 + 1, training=False)
    model = tf.keras.Model(inputs, outputs)
    # model.summary()

    # model.save_weights('models/test/test')
    model.load_weights('models/test/test')

    print(model.get_layer('quater').get_weights()[0][0][0][0:4])

I got this error: image

How can I fix this?

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

closed time in 22 days

murdockhou

issue commenttensorflow/tensorflow

model restore error/confict in tensorflow

This is happening because model.save/load_weights creates a tf.train.Checkpoint which assumes the models have the same structure. If you're changing the model structure between saving and loading, you should create a tf.train.Checkpoint object that is compatible with both models.

For example, if the layer names are consistent, then you could create a tf.train.Checkpoint that maps layer names to layers:

def create_checkpoint(model):
  return tf.train.Checkpoint(**{layer.name: layer for layer in model.layers})

model = ...
ckpt = create_checkpoint(model)
ckpt_path = ckpt.save("/path/to/ckpt")

# loading
model = ...
ckpt = create_checkpoint(model)
ckpt.restore(ckpt_path)
murdockhou

comment created time in 22 days

pull request commenttensorflow/tensorflow

Update save.py

Oops, another change is needed to push this.

Can you modify the line from tensorflow.python.keras.engine import network in save.py to use the LazyLoader? An example is here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/saving/saved_model/save_impl.py#L45

nikochiko

comment created time in a month

pull request commenttensorflow/estimator

Custom export outputs for keras estimator

Thanks for the PR! How would the user use this argument to specify custom export outputs?

wsuchy

comment created time in a month

issue commenttensorflow/tensorflow

Incompatible checkpoint being restored on estimator with keras model

I ran this and you're right, the restored_shapes is None. Looks like it is being set to None here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/saver.py#L336

So fixing this would just require creating savers with reshape=True.

kurka

comment created time in 2 months

issue commenttensorflow/tensorflow

Incompatible checkpoint being restored on estimator with keras model

tf.train.Saver uses RestoreV2/RestoreTensorsV2 instead of assign ops. Is this a bug with the restore ops?

kurka

comment created time in 2 months

issue commenttensorflow/tensorflow

Incompatible checkpoint being restored on estimator with keras model

All the checkpoint values are being loaded, but as a result the variable shapes are changed. It appears to be an unintentional bug from tf.train.Saver, not just a quirk of Estimator or Keras. The restore op doesn't do any shape checking. We should add a warning to saver.restore if any of the variables are reshaped.

kurka

comment created time in 2 months

issue commenttensorflow/tensorflow

Error when loading SavedModel with models.load_model() and compile=false

@lsgrep Feel free to pick this up if you're interested!

adrigrillo

comment created time in 2 months

pull request commenttensorflow/tensorflow

replaced builtin `all` by `np.all`

Thanks!

KristianHolsheimer

comment created time in 2 months

issue commenttensorflow/tensorflow

Saving custom tensor fails

This is an interesting use case. SavedModel should be correctly handling this, but we hadn't considered this case yet. I'll see what changes are required to resolve the issue.

awav

comment created time in 2 months

issue commenttensorflow/tensorflow

Inconsistent loading of saved model

There are issues with using HDF5 with GCS. I'm not sure when we'll be able to get to this, but please use the workaround where the model is copied locally before loading.

fonnesbeck

comment created time in 2 months

issue commenttensorflow/tensorflow

Can't save a Model with a TimeDistributed layer wrapping another Model

There appears to be a problem with saving layer masks using SavedModel. looking into this

artemmavrin

comment created time in 2 months

issue commenttensorflow/tensorflow

Output names lost when loading Keras model in SavedModel format

Confirming, thanks!

cstruct

comment created time in 2 months

issue closedtensorflow/tensorflow

Output names lost when loading Keras model in SavedModel format

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux-5.0.0-25-generic-x86_64-with-Ubuntu-18.04-bionic
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.0.0
  • Python version: 3.7.4

Describe the current behavior When loading a Keras model saved in the SavedModel format the output names are lost. Losing the output names causes loading to fail if using dictionaries for configuring loss or metrics.

Describe the expected behavior Output names to be restored when loading the model and dictionaries for losses to be working when loading the model.

Code to reproduce the issue output_names is not restored and outputs are given new auto-generated names:

import tensorflow as tf

i = tf.keras.layers.Input((1,))
x = tf.keras.layers.Dense(1, name='my-output')(i)
m = tf.keras.Model(i, x)
m.compile(loss='mse')

m.save('my-saved-model')
m2 = tf.keras.models.load_model('my-saved-model')
assert m2.output_names[0] == 'my-output'  # AssertionError

Model fails to load when using dictionaries for losses:

import tensorflow as tf

i = tf.keras.layers.Input((1,))
x = tf.keras.layers.Dense(1, name='my-output')(i)
m = tf.keras.Model(i, x)
m.compile(loss={'my-output': 'mse'})
assert m.output_names[0] == 'my-output'

m.save('my-saved-model')
m2 = tf.keras.models.load_model('my-saved-model') # ValueError: Unknown entries in loss dictionary: ['my-output']. Only expected following keys: ['output_1']

closed time in 2 months

cstruct

issue commenttensorflow/tensorflow

[tf-nightly] unable to load saved functional model

Thanks for reporting this Cospel. Confirming that this is a bug, I am in the middle of redoing SavedModel loading behavior which will address this issue.

Cospel

comment created time in 2 months

Pull request review commenttensorflow/tensorflow

Update save.py

 def save_model(model,   """   from tensorflow.python.keras.engine import sequential  # pylint: disable=g-import-not-at-top -  default_format = 'tf' if tf2.enabled() else 'h5'-  save_format = save_format or default_format--  if (save_format == 'h5' or-      (h5py is not None and isinstance(filepath, h5py.File)) or-      os.path.splitext(filepath)[1] in _HDF5_EXTENSIONS):+  if type(filepath) != str and not isinstance(filepath, h5py.File):

Perhaps refactor out this part of the code to a separate function that is called by both save_weights() and save().

Also, it would be useful to have unit tests to make sure that the save format returned by the function is correct

nikochiko

comment created time in 3 months

Pull request review commenttensorflow/tensorflow

Update save.py

 import six  from tensorflow.python import tf2+from tensorflow.python.keras.engine.network import _is_hdf5_filepath

Prefer importing the module instead of individual functions.

(You might have to disable the pylint check, see https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/engine/training.py#L231)

nikochiko

comment created time in 3 months

issue commenttensorflow/tensorflow

save method shows buggy/confusing behaviour

That would be great! Reviewing the PR now

nikochiko

comment created time in 3 months

pull request commenttensorflow/tensorflow

Allow `pathlib.Path` paths for loading models via Keras API

Thanks for the clear explanation, that makes sense.

mpetroff

comment created time in 3 months

pull request commenttensorflow/tensorflow

Allow `pathlib.Path` paths for loading models via Keras API

Hi, thanks for this change! Is here a reason why the user would prefer passing a pathlib.Path object to load_model, rather than calling load_model(str(obj)) themselves?

mpetroff

comment created time in 3 months

Pull request review commenttensorflow/community

RFC: TPU SavedModel Export API for TF2

+# TPU SavedModel Export API for TF2.x++| Status        | Proposed                                                |+| :------------ | :------------------------------------------------------ |+| **RFC #**     | [NNN](https://github.com/tensorflow/community/pull/NNN) |+:               : (update when you have community PR #)                   :+| **Author(s)** | Zhuoran Liu (lzr@google.com), Youlong Cheng (ylc@google.com) |+| **Sponsor**   | Jonathan Hseu (jhseu@google.com)                        |+| **Updated**   | 2019-11-06                                              |++## Objective++Provide an API to allow TF2 users to export TPU saved models <b>for+inference</b>, which:+++   Provide a user-friendly way to specify which function to run on TPU;++   Hides Graph construction and TPU inference specific logic (multi-core+    support, etc) from users;++   Allows specifying tags in SavedModel.++## Motivation++### Limitation of current `tf.saved_model.save()`++MetaGraphDef allows saving customized tags. Current downstream components like+TPU model-server, TFX infra-validator use the tags to load the specific+MetaGraph. However tf.saved_model.save() does not allow users to specify the set+of tags in MetaGraphDef, but hard-coded the MetaGraph to have only one ‘serve’+tag.++### Special Logic in TPU Inference Graph++Under the status quo, TPU computations have to be represented by a drastically+different graph from CPU. Inference-specific requirements (e.g. batching /+core-selection) also adds another layer of complexity.++Some major differences between CPU and TPU Graph:+++   As a protocol between TensorFlow Graph and TF2XLA, TPU device placement of a+    Node is done by attaching `_tpu_replicate` attribute;++   For multicore efficiency, TPU computations have to be encapsulated as a+    function and saved in FunctionLibrary, and will be called by+    TPUPartitionedCall. A TPUOrdinalSelector node has to be connected to+    TPUPartitionedCall to do efficient round-robin core selection;++   Variable nodes have to be lifted from TPU functions, rewritten as+    VarHandleOp, and consumed by ReadVariableOp.++Also for reducing the number of TPU compilation, serving platforms(For example,+Servomatic) prefers batching the inference requests with a few allowed batch+sizes. This requires wrapping TPUPartitionedCall in another function, and called+by BatchFunction.++Below is an intuitive example of how a TPU graph is different from a CPU one:++![Original CPU Graph](20191106-tf2-tpu-savedmodel/cpu_graph.png)+<center>Original CPU Graph.</center>++![TPU Graph](20191106-tf2-tpu-savedmodel/tpu_graph.png)+<center>TPU Graph.</center>++### User Control of Device Placement++There has to be a way for users to specify which part of computation should be+placed on TPU, because there’s no perfect device placement policy that can work+for every use case. For example even though dense embedding ops are allowed on+TPU, serving models might still want to run embedding lookups on CPU because the+embeddings are too big to fit on TPU.++![Customized Embeddings](20191106-tf2-tpu-savedmodel/customized_embeddings.png)+<center>Example of user control. In this graph, both ‘custom_embedding’ and+‘dense’ can run on TPU. But users want ‘custom_embedding’ to run on CPU for+whatever reason, e.g. CPU computations can be parallelized, users don’t have+enough TPU resources, etc. In this case, there has to be a way for them to tell+SavedModel that only ‘dense’ is to run on TPU.</center>++## User Benefit++Enable TPU Inference.++## Design Proposal++### User Facing API++<b>For General TF2 Users</b>++Users need to do the following things to export a TPU SavedModel in TF2.x:++1.  Replace @tf.function with @tf.tpu.function for functions they wish to run on+    TPU;++    ```python+    @tf.tpu.function+    def predict_step(images):+      ...+    ```++2.  Create main serving function and call the tpu function above. The main+    function might have additional TF ops which can’t run on TPU (e.g.+    `tf.decode_image`:++    ```python+    @tf.function+    def serve(images):+      image_tensors = tf.decode_image(images)+      return predict_step(image_tensors)+    ```++    And then create a signature:++    ```python+    signatures = {+        'serving_default':+            serve.get_concrete_function(...),+    }+    tags = [tag_constants.SERVING, tag_constants.TPU]+    ```++3.  Pass the both signatures to `tf.saved_model.save()`:++    ```python+    tf.saved_model.save(+         model,+         export_dir='...',+         signatures=signatures,+         tags=tags)+    ```++The resulting TPU inference graph looks like this:++![Resulting TPU Graph](20191106-tf2-tpu-savedmodel/tpu_result.png)+<center>Resulting TPU Graph.</center>++<b>For Advanced Users who need customized Ops</b>++In such cases, we provide the flexibility for users to tweak `@tf.tpu.function`.++1.  If users wish not to use TPUPartitionedCall, they can disable using+    TPUPartitionedCall:++    ```python+    @tf.tpu.function(use_tpu_partitioned_call=False)+    def predict_step(images):+      ...+    ```++2.  Users can nest TPU functions within BatchFunction:++    ```python+    @batch_ops.nondifferentiable_batch_function+    @tf.tpu.function+    def predict_step(images):+      ...+    ```++3.  User can also use their customized PartitionedCallOp:++    ```python+    @batch_ops.nondifferentiable_batch_function+    @my_partitioned_call_op_constructor+    @tf.tpu.function(use_tpu_partitioned_call=False)+    def predict_step(images):+      ...+    ```++<b>For Keras Users</b>++Keras users only need to pass `export_to_tpu=True` to save to TPU SavedModel.+(Currently, we require the Keras model being saved to be completely+TPU-compatible.)++```python+tf.keras.models.save_model(+    model,+    filepath='...',+    export_to_tpu=True)

If it's possible on the dist strat side, I prefer this idea over adding new arguments to the function.

lzr-google

comment created time in 3 months

issue commenttensorflow/tensorboard

Make it easy to visualize SavedModels with TensorBoard

Hello, what is the status of this?

gargn

comment created time in 4 months

issue commenttensorflow/tensorflow

Tensorflow 2.0.0 / tf.keras.layers.TimeDistributed layer can't be save to saved Model

This appears to be an error with Python 3. Submitting a fix to resolve this.

luvwinnie

comment created time in 4 months

issue commenttensorflow/tensorflow

[TF2.0] EstimatorV2 uses non existing export_savedmodel method

Estimator is supported in TF2, but the APIs for building the graph have been removed from the V2 API to keep the new API compatible with eager. You can continue to use Estimators but you'll have to use the compat.v1 for certain APIs like placeholder.

There are instructions here for updating your model to use the Keras API: https://www.tensorflow.org/guide/migrate#converting_models

Do you mean in Keras' ModelCheckpoint class? Yes, there're currently isn't a way to specify the keep the top N checkpoints. Feel free to create a new issue asking for a feature request.

MMMarcy

comment created time in 4 months

issue commenttensorflow/tensorflow

[TF2.0] EstimatorV2 uses non existing export_savedmodel method

@laetitiaoist Estimator still relies on graph-based code, so you would have to use tf.compat.v1.placeholder.

We're recommending that users switch to tf.Keras in the future, which has built-in support for eager mode.

MMMarcy

comment created time in 4 months

issue commenttensorflow/tensorflow

ValueError from invalid weights while loading older .h5 model

Looks like the renorm_stddev_weight variable was removed between 1.14 and 1.15 (https://github.com/tensorflow/tensorflow/commit/af15fb8624a0c0eabdd00ba1653cbbd4734c3b36). Accidentally breaking checkpoints is extremely easy. I would recommend that in the future, use the TensorFlow checkpoints (by specifying save_format=tf), which are more robust when dealing with variable changes.

You can migrating the Savefile following these steps (requires using different versions of TensorFlow).

Using TensorFlow 1.13 (or 1.14):

  • Load the model using model = tf.keras.models.load_model
  • Save the model into two separate files:
    • model.save_weights('model_weights.ckpt', save_format='tf') The weights file will still contain the renorm_stddev_weight variable, but will be omitted when loading.
    •   with open('model_config.json', 'w') as f:
          f.write(model.to_json()) 
      

Using TensorFlow 1.15:

  • Load the model:
    model = tf.keras.models.model_from_json(open('model_config.json').read())
    model.load_weights('model_weights.ckpt')
    
    The batch normalization layer will not have the renorm_stddev_weight variable, but the rest of the variable values will be restored.
  • Now save the model by calling model.save('model.h5')

@bcoopers I'm not super familiar with the BatchNormalization layer. I'm a bit concerned about correctness since a variable was entirely removed. If a model were trained using the renorm_stddev_weight, and then loaded without it, would it still function correctly?

ghannum

comment created time in 5 months

IssuesEvent

issue commenttensorflow/tensorflow

[TF2.0] Checkpoint doesn't store non-TF objects

When the values were restored, the list no longer contained any variables. Checkpoint.restore only restores the values of existing objects, but will not modify the objects themselves. In other words, it will not rewrite the elements in the list.

What would be the use case for storing python ints/floats/strings?

awav

comment created time in 5 months

issue closedtensorflow/tensorflow

[TF2.0] Checkpoint doesn't store non-TF objects

Hello all,

I found that tf.train.Checkpoint doesn't save non-TF objects and tf.Tensors. The MWE below shows the case:

import tensorflow as tf

class A(tf.Module):
    def __init__(self):
        super().__init__()
        self.scalar = 1.
        self.variable = tf.Variable(1.)
        self.tensor = tf.convert_to_tensor(1.)
        self.list = [tf.convert_to_tensor(10.0), 20.0, tf.Variable(0.0)]

    def __str__(self):
        return f"scalar={self.scalar}, variable={self.variable.numpy()}, tensor={self.tensor}, list={self.list}"


a_module = A()
checkpoint = tf.train.Checkpoint(a=a_module)
manager = tf.train.CheckpointManager(checkpoint, '/tmp/example', max_to_keep=3)
manager.save()

print(f"1. {a_module}")

#### modify

a_module.scalar = -100.
a_module.variable.assign(123.)
a_module.tensor = tf.convert_to_tensor(-12.)
a_module.list = [3., 3.]

print(f"2. {a_module}")

#### restore
checkpoint.restore(manager.latest_checkpoint)

print(f"3. {a_module}")

Output:

1. scalar=1.0, variable=1.0, tensor=1.0, list=ListWrapper([<tf.Tensor: id=9, shape=(), dtype=float32, numpy=10.0>, 20.0, <tf.Variable 'Variable:0' shape=() dtype=float32, numpy=0.0>])
2. scalar=-100.0, variable=123.0, tensor=-12.0, list=ListWrapper([3.0, 3.0])
3. scalar=-100.0, variable=1.0, tensor=-12.0, list=ListWrapper([3.0, 3.0])

In the output you can see that the checkpointer restores only the variable and completely ignores other fields of the A object.

I expected that at least pythonic objects, numpy and TF tensors should be stored by the checkpointer on the disk. Is that a normal behavior?

If the answer is yes: I think that this feature should be implemented, otherwise it is very confusing and not straightforward why the checkpoint saves some objects and others not. no: this is a bug, must be fixed.

System information

  • OS Platform and Distribution: macOS 10.14.6
  • TensorFlow version: v1.12.1-10419-g5138353309 2.0.0-dev20190905
  • Python version: 3.6.8

closed time in 5 months

awav

issue commenttensorflow/tensorflow

[TF2.0] Checkpoint doesn't store non-TF objects

Hi @awav,

This is intended. Tensorflow checkpoints only save and restore the variables/other TensorFlow objects with state. Constants tensors and python objects won't be included in the Checkpoint, and this is expected because some constants (e.g. lookup tables) can get exceedingly large.

There is an experimental API that lets you save arbitrary python values to the Checkpoint. Check it out here: https://www.tensorflow.org/api_docs/python/tf/train/experimental/PythonState

Feel free to submit a PR with clarifications to docstring of tf.train.Checkpoint.

awav

comment created time in 5 months

issue commenttensorflow/tensorflow

Subclassing tf.keras.models.Model save_model saves h5py but not h5/hdf5 file types

Specifying that the SavedModel format is able to handle custom models sounds reasonable and would remove the ambiguity. I'll add that to the warning message. Thanks for the suggestion!

Ryandry1st

comment created time in 5 months

IssuesEvent

issue commenttensorflow/tensorflow

Subclassing tf.keras.models.Model save_model saves h5py but not h5/hdf5 file types

The NotImplementedError error from an earlier post above states that HDF5 cannot save out subclassed models. The SavedModel format is able to handle these models, so you should not get an error.

Ryandry1st

comment created time in 5 months

issue commenttensorflow/tensorflow

Subclassing tf.keras.models.Model save_model saves h5py but not h5/hdf5 file types

I would like to get a better understanding of the issue is.

Perhaps docstring for the save_format is unclear? Currently, it says:

save_format: Either 'tf' or 'h5'. A filepath ending in '.h5' or '.keras' will default to HDF5 if save_format is None. Otherwise None defaults to 'tf'.
Ryandry1st

comment created time in 5 months

issue closedtensorflow/tensorflow

Subclassing tf.keras.models.Model save_model saves h5py but not h5/hdf5 file types

<em>Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em>

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab
  • TensorFlow installed from (source or binary): Pip
  • TensorFlow version (use command below): 2.0.0b
  • Python version: 3.6.7
  • GPU model and memory: Tesla T4

Describe the current behavior -Using tf 2.0.0b-gpu on google colab.

While using the subclassing API for a subclassed layer and model, I was unable to use the model.save_model() function for h5 or hdf5 file types, but I could successfully save and load the model if it was saved as h5py file type. In the toy example being used it worked correctly, although this may not be the case. Note that the get_config method was implemented in the custom layer and custom model.

Describe the expected behavior Either the save_model should always work (I believe this is a feature goal) and the documentation should reflect this, or if the save is likely to produce incorrect results it should raise an error and the documentation should continue to suggest that custom models can only be saved with the save_weights feature.

Code to reproduce the issue

import tensorflow as tf
from tensorflow import keras

class resblock(keras.layers.Layer):
    def __init__(self, n_layers, n_neurons, **kwargs):
        super().__init__(**kwargs)
        self.hidden = [keras.layers.Dense(n_neurons,
                                          activation='elu',
                                          kernel_initializer='he_normal')
                       for _ in range(n_layers)]
        
    def call(self, inputs):
        z = inputs
        for layer in self.hidden:
            z = layer(z)
        return inputs + z
    
    def get_config(self):
        base_config = super().get_config()
        return {**base_config}

class res_mod(keras.models.Model):
    def __init__(self, output_dim, activation=None, **kwargs):
        super().__init__(**kwargs)
        self.f1 = keras.layers.Flatten()
        self.hidden1 = keras.layers.Dense(100, activation='elu', kernel_initializer='he_normal')
        self.b1 = resblock(2, 100)
        self.b2 = resblock(2, 100)
        self.output1 = keras.layers.Dense(output_dim, activation=keras.activations.get(activation))
        
    def call(self, inputs):
        z = self.f1(inputs)
        z = self.hidden1(z)
        for _ in range(4):
            z = self.b1(z)
        z = self.b2(z)
        return self.output1(z)
    
    def get_config(self):
        base_config = super().get_config()
        return{**base_config, "output_dim" : output_dim, "activation": activation}
    

model = res_mod(10, activation='softmax')
model.compile(loss='sparse_categorical_crossentropy', optimizer='nadam', metrics=['accuracy'])

model.fit(train, epochs=25, validation_data=test)
# This is able to save and works correctly, returning the trained model
model.save('custom_model.h5py')
del model
model = keras.models.load_model('custom_model.h5py', custom_objects={'resblock': resblock})

Other info / logs This will raise an error that only sequential or functional models can be saved model.save('custom_model.h5')

closed time in 5 months

Ryandry1st

issue commenttensorflow/tensorflow

Subclassing tf.keras.models.Model save_model saves h5py but not h5/hdf5 file types

Calling model.save('custom_model.h5py') still saves out in the SavedModel format, even though the extension says .h5py (so it's not actually using h5py).

Ryandry1st

comment created time in 5 months

issue commenttensorflow/tensorflow

TypeError: 'Not JSON Serializable' while doing tf.keras.Model.save and using keras variable in loss_weights in tf.keras.Model.compile

Are you saving just the weights or the model? If just the weights, then model.save_weights should suffice.

tejal567

comment created time in 5 months

issue commenttensorflow/tensorflow

TypeError: 'Not JSON Serializable' while doing tf.keras.Model.save and using keras variable in loss_weights in tf.keras.Model.compile

@vsatyakumar Currently, you can't use tensorflow variables objects in the compiled args. I'll see if there is a fix for this. For now, a workaround would be to save the uncompiled version of the model, then manually re-compile once its loaded.

@hno1993 I believe if you wrapped the encoder ops (shown below) as a custom layer, the serialization would work.

 embeddings = tf.keras.layers.Embedding(hparams.vocab_size,
                                         hparams.d_model)(inputs)
  embeddings *= tf.math.sqrt(tf.cast(hparams.d_model, tf.float32))
  embeddings = PositionalEncoding(hparams)(embeddings)

  outputs = tf.keras.layers.Dropout(hparams.dropout)(embeddings)

Your code may be buggy if you're running in eager mode, since tf.keras.layers.Embedding(...)(inputs) would immediately evaluate the variables in the Embedding layer to constant values.

tejal567

comment created time in 5 months

issue closedtensorflow/tensorflow

skip_mismatch not supported in load_weights for Keras model

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS
  • TensorFlow installed from (source or binary): pip
  • TensorFlow version (use command below): 1.13.1
  • Python version: 3.7.3

Describe the current behavior I have noticed that mixing-and-matching keras and tensorflow.keras imports can cause some issues. I am therefore in the process of changing all keras imports in my projects to tensorflow.keras imports. I have now stumbled on the following problem: The function load_weights for a keras model does support the argument skip_mismatch in keras, but not in tensorflow.keras. Is this feature in the making?

Describe the expected behavior The tensorflow.keras function load_weights to have the same syntax as Keras's.

Code to reproduce the issue

# Behavior depends on the import
# from tensorflow.keras import *
# from keras import *

model = models.Sequential()
model.load_weights(None, by_name=True, skip_mismatch=True)

Note that this code fails in any case, however, only with the tf.keras import this is because of invalid parameters.

closed time in 5 months

kriskorrel-cw

issue commenttensorflow/tensorflow

skip_mismatch not supported in load_weights for Keras model

Thanks for the request, this argument has been added. You should see the changes in the next nightly or TF 2.1.

kriskorrel-cw

comment created time in 5 months

issue commenttensorflow/tensorflow

"TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO" when trying to restore a model from a byte stream in tensorflow 2.0.0-beta0

I tried running the colab with tensorflow==2.0.0-alpha0, but the model was still unable to be loaded. load_model's docstring states that it only supports:

filepath: One of the following:
          - String, path to the saved model
          - `h5py.File` object from which to load the model

Can you try passing the h5 File object to load_model instead?

helloIAmPau

comment created time in 5 months

issue commenttensorflow/tensorflow

Model unable to take input from dictionary(with input layer names as key) after loading it using tf.keras.models.load_model

I believe that this is fixed with commit: https://github.com/tensorflow/tensorflow/commit/6a26d679113105820cf83a2447863a6a32488c47#diff-29e6348914dee326daf67d86aa5075db.

Ran the colab (thanks @ravikyram!) with tf-nightly-2.0-preview, and it worked.

Shubham3101

comment created time in 5 months

issue closedtensorflow/tensorflow

Model unable to take input from dictionary(with input layer names as key) after loading it using tf.keras.models.load_model

System information

  • OS Platform and Distribution (Linux Ubuntu 16.04): device:
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.0 rc
  • Python version: 3.6

My model works fine the first time when i create the architecture and train it, but once i save the model using tf.keras.model.save_model and then load it again using tf.keras.models.load_model it fails to take input from the dictionary(with input layer names as key) as if the names of layers are changed.

Example: training_mask = tf.keras.layers.Input(shape=(None, None, 1), name='training_mask') target_score_map = tf.keras.layers.Input(shape=(None, None, 1), name='target_score_map') target_geo_map = tf.keras.layers.Input(shape = (None,None,5), name='target_geo_map')

These are three input layers that i am using to take input for training but once i save the model and later load it again to train it gives me the error " No data provided for "input_1". Need data for each key in: ['input_1', 'input_2', 'input_3']"

I am feeding data by specifying layer names :

    inputs = {
            'target_score_map': score_maps,
            'target_geo_map': geo_maps,
            'training_mask': training_masks
        }

Therefore, it works the first time when create I the architecture from code then train and save it but later when i load the model from disk it fails.

Regards Shubham

closed time in 5 months

Shubham3101

PR opened tensorflow/tensorflow

Reviewers
Fix bug when cloning functional models that use Tensor keyword argume…

…nts.

Aligned the cloning implementation to be similar to Model.from_config(model.get_config()), with a few minor differences.

PiperOrigin-RevId: 268969706

+356 -263

0 comment

3 changed files

pr created time in 5 months

create barnchk-w-w/tensorflow

branch : cherrypicks_9FILR

created branch time in 5 months

issue closedtensorflow/tensorflow

tf.keras.models.load_model takes forever to load

System information

  • Have I written custom code: yes
  • OS Platform and Distribution: Windows 10 Version 1607
  • TensorFlow installed from: pip
  • TensorFlow version:
    • Git-Version: v1.12.1-8794-ge36271a61d
    • version: 1.15.0-dev20190814
  • CUDA/cuDNN version: 10.0/7
  • GPU model and memory: NVIDIA GTX 1050 Ti, 4096 MB

Describe the current behavior The code below produces this output:

Save successfull
loading ...
W0815 15:02:11.705600 17844 deprecation.py:506] From C:\tensorflow_anduin\lib\site-packages\tensorflow_core\python\ops\init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0815 15:02:11.706604 17844 deprecation.py:506] From C:\tensorflow_anduin\lib\site-packages\tensorflow_core\python\ops\init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will
be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

even after 30 mins it doesn't print the expected output Load successfull

Describe the expected behavior Succesful loading in under 1 min

Code to reproduce the issue

import tensorflow as tf
from tensorflow.keras.layers import (Dense, Input, Lambda)
from tensorflow.keras.models import Model, Sequential
from scipy import sparse
import numpy as np


def layer_lambda(input_x):
    sparse = input_x[0]
    dense = input_x[1]
    dense = tf.transpose(dense)
    y = tf.sparse.sparse_dense_matmul(sparse, dense)
    return tf.transpose(y)


dense_mat = np.eye(30, 30, dtype=np.float32)
sparse_mat = sparse.coo_matrix(dense_mat)
sparse_indices = np.mat([sparse_mat.row, sparse_mat.col]).transpose()
sparse_tensor = tf.SparseTensor(sparse_indices, sparse_mat.data, sparse_mat.shape)

model = Sequential()
model_input = Input(shape=(20,))
x = Dense(20)(model_input)
x = Dense(30)(x)
x = Lambda(layer_lambda, output_shape=(None, 30, 30))([sparse_tensor, x])
model = Model(model_input, x)

model.predict([[np.ones(20)]])

model.save("model.h5")

print("Save successfull")
print("loading ...")
model_load = tf.keras.models.load_model("model.h5", custom_objects={'layer_lambda': layer_lambda})

print("Load successfull")

This error occurred after fixing issue 31607.

closed time in 5 months

Asorie

issue commenttensorflow/tensorflow

tf.keras.models.load_model takes forever to load

Moving all of the sparse ops into the lambda solves this issue:

def layer_lambda(input_x):
  dense_mat = np.eye(30, 30, dtype=np.float32)
  sparse_mat = sparse.coo_matrix(dense_mat)
  sparse_indices = np.mat([sparse_mat.row, sparse_mat.col]).transpose()
  sparse_tensor = tf.SparseTensor(sparse_indices, sparse_mat.data, sparse_mat.shape)
  dense = input_x
  dense = tf.transpose(dense)
  y = tf.sparse.sparse_dense_matmul(sparse_tensor, dense)
  return tf.transpose(y)

model = Sequential()
model_input = Input(shape=(20,))
x = Dense(20)(model_input)
x = Dense(30)(x)
x = Lambda(layer_lambda, output_shape=(None, 30, 30))(x)
model = Model(model_input, x)
Asorie

comment created time in 5 months

create barnchk-w-w/tensorflow

branch : cherrypicks_1LOIZ

created branch time in 5 months

issue commenttensorflow/tensorflow

[TF2.0] EstimatorV2 uses non existing export_savedmodel method

The issue in the OP with the wrong use of export_savedmodel should be fixed in 1.14.

@tarrade That looks like a different bug, can you create a new issue with some code to reproduce the errors? Although from the code snippet, you should be creating a placeholder and not a variable. If you're using a variable, the initialization arguments are in a different order.

MMMarcy

comment created time in 5 months

issue closedtensorflow/tensorflow

[TF2.0] EstimatorV2 uses non existing export_savedmodel method

Dear tensorflowers,

as per the title, EstimatorV2's exporter.py calls a method that has been removed. The fix is pretty easy, but I'm not sure about the implications of the change. See more below.

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): custom code, but using estimators
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 4.19.20-1rodete1-amd64 #1 SMP Debian 4.19.20-1rodete1
  • TensorFlow installed from (source or binary): pip install --upgrade tensorflow==2.0.0alpha0
  • TensorFlow version (use command below): 2.0.0-alpha0
  • Python version: 3.7.1

Describe the problem

using tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec) results in AttributeError: 'EstimatorV2' object has no attribute 'export_savedmodel' being thrown.

2 small modifications to my local tensorflow_estimator/python/estimator/exporter.py file fix the issue. However, before submitting a PR i wanted to clarify that I had to:

  1. Rename the method call to estimator.export_saved_model(...) and this poses no problem.
  2. Remove the strip_default_attrs=self._strip_default_attrs keyword argument, but I have no idea what this entails. I assume that, since you can't specify this argument anymore, it defaults to the default policy in TF 1.X of stripping the GraphDef default attributes. If that's the case, is there any action required?

Logs

  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 359, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1139, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1173, in _train_model_default
    saving_listeners)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1452, in _train_with_estimator_spec
    any_step_done = True
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 856, in __exit__
    self._close_internal(exception_type)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 889, in _close_internal
    h.end(self._coordinated_creator.tf_sess)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 588, in end
    self._save(session, last_step)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow/python/training/basic_session_run_hooks.py", line 607, in _save
    if l.after_save(session, step):
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 519, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 927, in evaluate_and_export
    is_the_final_export)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 960, in _export_eval_result
    is_the_final_export=is_the_final_export))
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/exporter.py", line 420, in export
    is_the_final_export)
  File "/usr/local/google/home/msteiner/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/exporter.py", line 120, in export
    export_result = estimator.export_savedmodel(
AttributeError: 'EstimatorV2' object has no attribute 'export_savedmodel'
Code

I'm trying to adapt the CMLE template to TF 2.0 . I avoided putting the input_fns and model_fns since the model trains and evals correctly. The only problem is during the export.

 train_input_fn = model_input.generate_input_fn(
      file_names_pattern=FLAGS.train_files,
      mode=tf.estimator.ModeKeys.TRAIN,
      num_epochs=FLAGS.num_epochs,
      batch_size=FLAGS.train_batch_size)

  eval_input_fn = model_input.generate_input_fn(
      file_names_pattern=FLAGS.eval_files,
      mode=tf.estimator.ModeKeys.EVAL,
      batch_size=FLAGS.eval_batch_size)



  if metadata.TASK_TYPE == "classification":
    estimator = model.create_classifier(config=run_config)
  elif metadata.TASK_TYPE == "regression":
    estimator = model.create_regressor(config=run_config)
  else:
    estimator = model.create_estimator(config=run_config)

  train_spec = tf.estimator.TrainSpec(
      train_input_fn,
      max_steps=int(train_steps),
  )

  # Export for the final prediction graph.
  exporter_default = tf.estimator.FinalExporter(
      'estimator',
      model_input.SERVING_FUNCTIONS[FLAGS.export_format](),
      as_text=False )

  eval_spec = tf.estimator.EvalSpec(
      eval_input_fn,
      steps=FLAGS.eval_steps,
      exporters=[exporter_default],  # This tells
      throttle_secs=FLAGS.eval_every_secs,
      start_delay_secs=0)

  # Main train and evaluate loop.
  tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

Happy to provide any additional information!

closed time in 5 months

MMMarcy

issue closedtensorflow/tensorflow

TF 2.0 save-load SequenceFeatures layer issue

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution: WIndows 10 Pro build 1903
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.0.0-alpha0, 2.0.0-beta0, 2.0.0-beta1, 2.0.0-rc0
  • Python version: Python 3.7.3

Describe the current behavior

  1. I create model using Keras Functional API. The model includes tf.keras.experimental.SequenceFeatures layer and sequence_categorical_column_with_vocabulary_list feature_column.
  2. Then I save model by tf.keras.models.model.save('path_to_model.h5').
  3. Finally, I load model by tf.keras.models.load_model(path_to_model.h5). While loading I get the error message :

Unknown layer: SequenceFeatures

Describe the expected behavior The model containing SequenceFeatures layer and any corresponding feature_columns is loaded by tf.keras.models.load_model(MODEL_PATH.h5) without errors.

Code to reproduce the issue

import tensorflow as tf
from tensorflow import feature_column

print(tf.__version__)

!python -V

# Define categorical colunm for our text feature, which is preprocessed into sequence of tokens
text_column = feature_column.sequence_categorical_column_with_vocabulary_list(key='text', 
                                                                     vocabulary_list=list(['asd', 'asdf']))

text_embedding = feature_column.embedding_column(text_column, dimension=8)

# Then define the layers and model it self
# This example uses Keras Functional API instead of Sequential just for more generallity

# Define SequenceFeatures layer to pass feature_columns into Keras model
sequence_feature_layer = tf.keras.experimental.SequenceFeatures(text_embedding)

max_length = 6
# Define inputs for each feature column. See
# см. https://github.com/tensorflow/tensorflow/issues/27416#issuecomment-502218673
sequence_feature_layer_inputs = {}
sequence_feature_layer_inputs['text'] = tf.keras.Input(shape=(max_length,), name='text', dtype=tf.string)

sequence_feature_layer_outputs, _ = sequence_feature_layer(sequence_feature_layer_inputs)
print(sequence_feature_layer_outputs)

# Define outputs of SequenceFeatures layer 
# And accually use them as first layer of the model

# note here that SequenceFeatures layer produce tuple of two tensors as output. We need just first to pass next.
sequence_feature_layer_outputs, _ = sequence_feature_layer(sequence_feature_layer_inputs)

x = tf.keras.layers.Conv1D(8,4)(sequence_feature_layer_outputs)
x = tf.keras.layers.MaxPooling1D(2)(x)
x = tf.keras.layers.Dense(256, activation='relu')(x)
x = tf.keras.layers.Dropout(0.2)(x)
x= tf.keras.layers.GlobalAveragePooling1D()(x)
# This example supposes binary classification, as labels are 0 or 1
x = tf.keras.layers.Dense(1, activation='sigmoid')(x)

model = tf.keras.models.Model(inputs=[v for v in sequence_feature_layer_inputs.values()], outputs=x)

model.summary()

# This example supposes binary classification, as labels are 0 or 1
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy']
              #run_eagerly=True
             )

model.save('model.h5')
model_loaded = tf.keras.models.load_model('model.h5')

model_loaded.summary()
# expected result: model_loaded is instantinated
# actual result: exception with text: "Unknown layer: SequenceFeatures"

#other way to save model structure also finishes with error

model_json_string = model.to_json()
# import pprint
# pprint.pprint(json.loads(model_json_string))
fresh_model = tf.keras.models.model_from_json(model_json_string)
fresh_model.summary()
# expected result: model_loaded is instantinated
# actual result: exception with text: "Unknown layer: SequenceFeatures"

Other info / logs I have found that this problem is fixed by changes in commit a29555a8fed1b2a1eb495e220eca7b3e3161ea26 and do not reproduced in tf-nightly-2.0-preview build. But it still reproduced in 2.0.0-alpha0, 2.0.0-beta0, 2.0.0-beta1, 2.0.0-rc0

closed time in 5 months

EgorBEremeev

issue commenttensorflow/tensorflow

TF 2.0 save-load SequenceFeatures layer issue

@EgorBEremeev The recommended format for saving is SavedModel. Can you try using that instead of Hdf5? To save to SavedModel, specify a path without the ".h5" extension, or set the save_format option to tf.

tf-nightly is more up to date than 2.0.0-alpha/beta/rc0, so the issue should be fixed in the next release.

EgorBEremeev

comment created time in 5 months

push eventk-w-w/tensorflow

Katherine Wu

commit sha f9494f24d8fc7f0bf451659c7eab0d48640410fb

Remove dedupe_weights @property Accidentally left it in the commit.

view details

push time in 6 months

pull request commenttensorflow/tensorflow

[r2.0-CherryPick]:Keras SavedModel important bug fixes and refactor

Closing this, new PR here: #32306

k-w-w

comment created time in 6 months

PR opened tensorflow/tensorflow

[r2.0-CherryPick]:Keras SavedModel important bug fixes and refactor
  1. Use of external tensors (e.g., generated by tf.sequence_mask) are converted to constants when saving.
  2. Fixed error preventing models with @tf.function decorated calls from being exported.
  3. Refactored Keras Serialization code (this change must be cherrypicked for the other two commits to be pushed)
+1241 -748

0 comment

16 changed files

pr created time in 6 months

create barnchk-w-w/tensorflow

branch : cherrypicks_UR7H3

created branch time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_XSBZI

delete time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_HH6F5

delete time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_WPCYV

delete time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_95NN9

delete time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_PTCRH

delete time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_RSDJN

delete time in 6 months

PR closed tensorflow/tensorflow

Reviewers
Add SaveOptions object with option to whitelist op namespaces. cla: yes comp:keras size:M

All saved_model saving functions now have an options argument where the user may pass in a SaveOptions object.

PiperOrigin-RevId: 266021878

+201 -27

0 comment

27 changed files

k-w-w

pr closed time in 6 months

PR closed tensorflow/tensorflow

Reviewers
[r2.0-CherryPick]:Keras SavedModel important bug fixes and refactor cla: yes size:XL
  1. Use of external tensors (e.g., generated by tf.sequence_mask) are converted to constants when saving.
  2. Fixed error preventing models with @tf.function decorated calls from being exported.
  3. Refactored Keras Serialization code (this change must be cherrypicked for the other two commits to be pushed)
+1244 -750

0 comment

16 changed files

k-w-w

pr closed time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_9TURB

delete time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_0J4QT

delete time in 6 months

delete branch k-w-w/tensorflow

delete branch : cherrypicks_66PJH

delete time in 6 months

create barnchk-w-w/tensorflow

branch : cherrypicks_PTCRH

created branch time in 6 months

push eventk-w-w/tensorflow

Reed Wanderman-Milne

commit sha 0fe04a9a523d291cc2658e6e2c5a108356bcb5ff

Add loss_scale field to Policy. A Keras Model will wrap it's optimizer with a LossScaleOptimizer, if it's policy has a loss scale. This way, users do not have to manually wrap their optimizers with a LossScaleOptimizer. PiperOrigin-RevId: 263186511

view details

Reed Wanderman-Milne

commit sha 75a9d9994115bb38af04b5e71aa6f324bc63b94e

Add mixed_float16 and mixed_bfloat16 dtype policies. These policies will be the recommended way of using mixed precision in tf.keras. So far, the only difference from [b]float16_with_float32_vars is that mixed_float16 enables dynamic loss scaling by default (and mixed_bfloat16 has no difference). In the future, the *_with_float32_vars policies will be removed. PiperOrigin-RevId: 263206151

view details

Reed Wanderman-Milne

commit sha a3a9fd34e98d361a4d9ae33f476887a8664f5e91

Do not cast inputs to AddLoss layers This means tensors passed to Model.add_loss will no longer be cast to floatx. PiperOrigin-RevId: 264287945

view details

Reed Wanderman-Milne

commit sha 6b9a66d5438c536f925ede3e1ce4699533f11e93

Deprecate "_with_float32_vars" policies. These policies will be removed in TensorFlow 2.1. I plan on removing them very shortly. PiperOrigin-RevId: 267429724

view details

Thomas O'Malley

commit sha 873bfb7b9176627d1e2e737902fc6ee18bd5fc35

Fix merge conflicts

view details

Reed Wanderman-Milne

commit sha 821b2fc9f10a59a21723dba5994dea194cfd5cce

Implement __repr__ for LossScale subclasses. PiperOrigin-RevId: 262603680

view details

Reed Wanderman-Milne

commit sha 43c6071ead8f9476606d1f087c1d0c1ed628cc7c

Improve Layer docstrings in regards to autocasting. I fixed the examples so they actually can run now. And I mentioned that currently, only the first argument to call() is casted. PiperOrigin-RevId: 262603558

view details

Penporn Koanantakool

commit sha a8d963daa290c80b0e017fa35652b2dcfc9bdd47

Rollforward of PR #32169: Upgrading giflib to fix CVE-2019-15133 Add a patch file to fix giflib's compilation issue on Windows (replace a call to strtok_r with strtok_s). **NVD**: 2019/08/17 - CVSS v2.0 Base Score: 4.3 - CVSS v3.0 Base Score: 6.5 In GIFLIB before 2019-02-16, a malformed GIF file triggers a divide-by-zero exception in the decoder function DGifSlurp in dgif_lib.c if the height field of the ImageSize data structure is equal to zero. Source | Link | Type ---- | ---- | ---- MISC | bugs.chromium.org | Mailing List, Third Party Advisory UBUNTU | usn.ubuntu.com | Third Party Advisory PiperOrigin-RevId: 267533902

view details

Goldie Gadde

commit sha 5a580681ad3e21c0e6622f15cb186154855eac60

Merge pull request #32257 from omalleyt12/cherrypicks_20IEW [r2.0-CherryPick] Deduplicate Keras weights

view details

Goldie Gadde

commit sha 8ea0a418a6efff8d2a0e063bbf51e0105910c040

Merge pull request #32269 from reedwm/mp_cherrypicks [r2.0-rc1 CherryPick]: Several tf.keras mixed precision API changes

view details

Goldie Gadde

commit sha d2e5f5a49ec79f9f6aa9720b38eb1ff4bddfa579

Merge pull request #32272 from reedwm/docstring_cherrypicks [r2.0-rc1 CherryPick]: Improve Layer docstrings in regards to autocasting.

view details

Goldie Gadde

commit sha b6e4c892679537694badf37db4b306dafc200bfc

Merge pull request #32297 from penpornk/cherrypicks_IF3LM [r2.0 CherryPick]: Upgrading giflib to fix CVE-2019-15133

view details

Katherine Wu

commit sha f86d8b128bfe1e533b4f802d95b7e72b7991de77

Merge branch 'r2.0' into cherrypicks_66PJH

view details

push time in 6 months

push eventk-w-w/tensorflow

Kathy Wu

commit sha dd293631025e17732a69adb0c8b5b21a274dccd0

Remove unused imports and resolve lint errors

view details

push time in 6 months

PR opened tensorflow/tensorflow

Reviewers
Keras SavedModel important bug fixes and refactor
  1. Use of external tensors (e.g., generated by tf.sequence_mask) are converted to constants when saving.
  2. Fixed error preventing models with @tf.function decorated calls from being exported.
  3. Refactored Keras Serialization code (this change must be cherrypicked for the other two commits to be pushed)
+1244 -749

0 comment

16 changed files

pr created time in 6 months

push eventk-w-w/tensorflow

Katherine Wu

commit sha 7b265c06b54bb05acfc2d0bcf06d97a927eb07bf

Remove SavedModel-specific methods from base Layer and Model, and add documentation for updating the serialization implementation. PiperOrigin-RevId: 266978650

view details

Katherine Wu

commit sha c0e0440aa743f45764ce3d43d615a70d3ba96995

Convert tensors to constants when saving Keras functional and graph models to SavedModel. This includes changes to backend.get_value() to ensure that the correct graph context created when getting the value of a tensor. PiperOrigin-RevId: 267498198

view details

Katherine Wu

commit sha 7ee62aa6b76a3205285a1e0489dccdc00f64f220

If layer's call function is wrapped in tf.function, then trace the original method instead of the the wrapped function. This avoids the issue where canonicalize_function_inputs added an additional defaults to the `args` list. PiperOrigin-RevId: 267650429

view details

push time in 6 months

create barnchk-w-w/tensorflow

branch : cherrypicks_66PJH

created branch time in 6 months

PR opened tensorflow/tensorflow

Reviewers
Add SaveOptions object with option to whitelist op namespaces.

All saved_model saving functions now have an options argument where the user may pass in a SaveOptions object.

PiperOrigin-RevId: 266021878

+201 -27

0 comment

27 changed files

pr created time in 6 months

create barnchk-w-w/tensorflow

branch : cherrypicks_0J4QT

created branch time in 6 months

more