profile
viewpoint

Ask questionsTFLite quantization silently converts bias tensor to int8 (int32 expected), causing interpreter to crash

System information

  • OS Platform and Distribution: Linux Ubuntu 20.04
  • TensorFlow installed from: source
  • TensorFlow version (or github SHA if from source): tf-nightly, version 2.4.0-dev20200721

Code I am converting using the options recommended in the docs:

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.target_spec.supported_ops = [
      tf.lite.OpsSet.TFLITE_BUILTINS_INT8
]
converter.target_spec.supported_types = [tf.int8, tf.uint8, tf.int32]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

# Allocating tensors subsequently fails
interpreter = tf.lite.Interpreter(model_content=tflite_model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]

The output from the converter invocation

2020-07-22 11:51:55.215847: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:192] None of the MLIR optimization passes are enabled (registered 0 passes)
INFO:tensorflow:The given SavedModel MetaGraphDef contains SignatureDefs with the following keys: {'serving_default'}
I0722 11:51:55.644716 140663456084352 convert_saved_model.py:80] The given SavedModel MetaGraphDef contains SignatureDefs with the following keys: {'serving_default'}
INFO:tensorflow:input tensors info: 
I0722 11:51:55.645021 140663456084352 convert_saved_model.py:99] input tensors info: 
INFO:tensorflow:Tensor's key in saved_model's tensor_map: input-to-image
I0722 11:51:55.645142 140663456084352 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: input-to-image
INFO:tensorflow: tensor name: Placeholder:0, shape: (1, 640, 640, 3), type: DT_FLOAT
I0722 11:51:55.645210 140663456084352 convert_saved_model.py:43]  tensor name: Placeholder:0, shape: (1, 640, 640, 3), type: DT_FLOAT
INFO:tensorflow:output tensors info: 
I0722 11:51:55.645267 140663456084352 convert_saved_model.py:101] output tensors info: 
INFO:tensorflow:Tensor's key in saved_model's tensor_map: raw_scores
I0722 11:51:55.645350 140663456084352 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: raw_scores
INFO:tensorflow: tensor name: RawScores:0, shape: (1, 76725, 4), type: DT_FLOAT
I0722 11:51:55.645412 140663456084352 convert_saved_model.py:43]  tensor name: RawScores:0, shape: (1, 76725, 4), type: DT_FLOAT
INFO:tensorflow:Tensor's key in saved_model's tensor_map: raw_boxes
I0722 11:51:55.645479 140663456084352 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: raw_boxes
INFO:tensorflow: tensor name: RawBoxes:0, shape: (1, 76725, 1, 4), type: DT_FLOAT
I0722 11:51:55.645537 140663456084352 convert_saved_model.py:43]  tensor name: RawBoxes:0, shape: (1, 76725, 1, 4), type: DT_FLOAT
2020-07-22 11:51:55.646430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 11:51:55.646453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      
INFO:tensorflow:Restoring parameters from /home/rsaini/repos/isr/edge-test/tmp/variables/variables
I0722 11:51:56.537452 140663456084352 saver.py:1293] Restoring parameters from /home/rsaini/repos/isr/edge-test/tmp/variables/variables
2020-07-22 11:51:57.199313: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2020-07-22 11:51:57.199419: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-07-22 11:51:57.343124: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x561781c5b260 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-22 11:51:57.343157: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1
2020-07-22 11:51:57.343674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7335GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2020-07-22 11:51:57.343771: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-22 11:51:57.343819: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:51:57.343863: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:51:57.343906: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:51:57.343949: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:51:57.343992: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:51:57.344035: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2020-07-22 11:51:57.344043: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-07-22 11:51:57.344057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 11:51:57.344064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-07-22 11:51:57.344070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-07-22 11:51:57.448338: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: graph_to_optimize
2020-07-22 11:51:57.448368: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: function_optimizer did nothing. time = 0.002ms.
2020-07-22 11:51:57.448373: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: function_optimizer did nothing. time = 0ms.
WARNING:tensorflow:From /home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/tensorflow/lite/python/util.py:276: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
W0722 11:51:57.642830 140663456084352 deprecation.py:323] From /home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/tensorflow/lite/python/util.py:276: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py:854: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
W0722 11:51:57.643322 140663456084352 deprecation.py:323] From /home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py:854: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2020-07-22 11:51:58.442749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 11:51:58.442781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      
INFO:tensorflow:Restoring parameters from /home/rsaini/repos/isr/edge-test/tmp/variables/variables
I0722 11:51:59.415019 140663456084352 saver.py:1293] Restoring parameters from /home/rsaini/repos/isr/edge-test/tmp/variables/variables
INFO:tensorflow:The given SavedModel MetaGraphDef contains SignatureDefs with the following keys: {'serving_default'}
I0722 11:51:59.910989 140663456084352 convert_saved_model.py:80] The given SavedModel MetaGraphDef contains SignatureDefs with the following keys: {'serving_default'}
INFO:tensorflow:input tensors info: 
I0722 11:51:59.911199 140663456084352 convert_saved_model.py:99] input tensors info: 
INFO:tensorflow:Tensor's key in saved_model's tensor_map: input-to-image
I0722 11:51:59.911319 140663456084352 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: input-to-image
INFO:tensorflow: tensor name: Placeholder:0, shape: (1, 640, 640, 3), type: DT_FLOAT
I0722 11:51:59.911397 140663456084352 convert_saved_model.py:43]  tensor name: Placeholder:0, shape: (1, 640, 640, 3), type: DT_FLOAT
INFO:tensorflow:output tensors info: 
I0722 11:51:59.911462 140663456084352 convert_saved_model.py:101] output tensors info: 
INFO:tensorflow:Tensor's key in saved_model's tensor_map: raw_boxes
I0722 11:51:59.912057 140663456084352 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: raw_boxes
INFO:tensorflow: tensor name: RawBoxes:0, shape: (1, 76725, 1, 4), type: DT_FLOAT
I0722 11:51:59.912123 140663456084352 convert_saved_model.py:43]  tensor name: RawBoxes:0, shape: (1, 76725, 1, 4), type: DT_FLOAT
INFO:tensorflow:Tensor's key in saved_model's tensor_map: raw_scores
I0722 11:51:59.912191 140663456084352 convert_saved_model.py:41] Tensor's key in saved_model's tensor_map: raw_scores
INFO:tensorflow: tensor name: RawScores:0, shape: (1, 76725, 4), type: DT_FLOAT
I0722 11:51:59.912250 140663456084352 convert_saved_model.py:43]  tensor name: RawScores:0, shape: (1, 76725, 4), type: DT_FLOAT
2020-07-22 11:51:59.913029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 11:51:59.913052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      
INFO:tensorflow:Restoring parameters from /home/rsaini/repos/isr/edge-test/tmp/variables/variables
I0722 11:52:00.961747 140663456084352 saver.py:1293] Restoring parameters from /home/rsaini/repos/isr/edge-test/tmp/variables/variables
2020-07-22 11:52:01.741302: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2020-07-22 11:52:01.741406: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-07-22 11:52:01.742209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7335GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2020-07-22 11:52:01.742323: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-22 11:52:01.742382: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:01.742436: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:01.742488: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:01.742540: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:01.742592: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:01.742644: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2020-07-22 11:52:01.742654: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-07-22 11:52:01.742672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 11:52:01.742680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-07-22 11:52:01.742688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-07-22 11:52:01.850868: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: graph_to_optimize
2020-07-22 11:52:01.850904: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: function_optimizer did nothing. time = 0.003ms.
2020-07-22 11:52:01.850913: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: function_optimizer did nothing. time = 0ms.
I0722 11:52:03.153840 140663456084352 lite.py:1321] Using experimental converter: If you encountered a problem please file a bug. You can opt-out by setting experimental_new_converter=False
2020-07-22 11:52:03.415416: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:315] Ignored output_format.
2020-07-22 11:52:03.415449: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:318] Ignored drop_control_dependency.
2020-07-22 11:52:03.791157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7335GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2020-07-22 11:52:03.791318: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-22 11:52:03.791376: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:03.791429: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:03.791480: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:03.791544: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:03.791592: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
2020-07-22 11:52:03.791640: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2020-07-22 11:52:03.791649: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-07-22 11:52:03.791666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 11:52:03.791673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-07-22 11:52:03.791679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-07-22 11:52:04.930987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 11:52:04.931019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      
2020-07-22 11:52:05.066647: W tensorflow/core/common_runtime/executor.cc:1086] [/device:CPU:0] Executor start aborting: Invalid argument: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [34600]
	 [[{{node Placeholder/_0}}]]
2020-07-22 11:52:05.066900: W tensorflow/core/common_runtime/executor.cc:1086] [/device:CPU:0] Executor start aborting: Invalid argument: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [34600]
	 [[{{node Placeholder/_0}}]]
Traceback (most recent call last):
  File "./export_tflite_model.py", line 91, in <module>
    tf.app.run(main)
  File "/home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "./export_tflite_model.py", line 85, in main
    export(FLAGS.saved_model_dir, FLAGS.output_dir)
  File "./export_tflite_model.py", line 79, in export
    interpreter.allocate_tensors()
  File "/home/rsaini/.pyenv/versions/edge/lib/python3.7/site-packages/tensorflow/lite/python/interpreter.py", line 243, in allocate_tensors
    return self._interpreter.AllocateTensors()
RuntimeError: tensorflow/lite/kernels/conv.cc:334 bias->type != kTfLiteInt32 (INT8 != INT32)Node number 3 (CONV_2D) failed to prepare.

Helpful Files Saved model: saved.zip

Graph after transformation: after-transformation.zip

Failure details Interestingly, the actual TFLiteConverter quantizes without crashing; reloading the generated model into the interpreter is what causes the crash. I am not sure why the bias tensor is being cast to int8; I am happy to provide any more debug files should they be necessary.

tensorflow/tensorflow

Answer questions renjie-liu

Can you try set experimental quantizer to be true and try again?

Thanks!

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/python/lite.py#L421

useful!

Related questions

ModuleNotFoundError: No module named 'tensorflow.contrib' hot 9
Tf.Keras metrics issue hot 8
Error occurred when finalizing GeneratorDataset iterator hot 7
Error loading tensorflow hot 6
module 'tensorflow' has no attribute 'ConfigProto' hot 6
TF 2.0 'Tensor' object has no attribute 'numpy' while using .numpy() although eager execution enabled by default hot 6
tensorflow-gpu CUPTI errors
Lossy conversion from float32 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning.
ModuleNotFoundError: No module named 'tensorflow.contrib'
When importing TensorFlow, error loading Hadoop
OSError: SavedModel file does not exist at: saved_model_dir/{saved_model.pbtxt|saved_model.pb}
AttributeError: module &#39;tensorflow.python.framework.op_def_registry&#39; has no attribute &#39;register_op_list&#39;
tf.keras.layers.Conv1DTranspose ?
[TF 2.0] tf.keras.optimizers.Adam hot 4
TF2.0 AutoGraph issue hot 4
source:https://uonfu.com/
Github User Rank List