profile
viewpoint

Ask questionstf.range + for x,y in dataset issue

System information

  • Have I written custom code: yes
  • OS Platform and Distribution: win10 1809, pycharm2019.2.3
  • TensorFlow installed from: pip install tensorflow-gpu
  • TensorFlow version: tensorflow-gpu 2.0.0
  • Python version: 3.6
  • CUDA/cuDNN version: 10.0/7.6.4
  • GPU model and memory: rtx2080 8g

Describe the current behavior tf.range + for x,y in dataset raise exception.

Describe the expected behavior run normally

Code to reproduce the issue

import tensorflow as tf

if __name__ == '__main__':
    dataset = tf.data.Dataset.from_tensor_slices(([1, 2, 3], [4, 5, 6]))

    @tf.function
    def f():
        for e in tf.range(3):
            for x, y in dataset:
                tf.print(x, y, e)
    f()

Other info / logs

2019-11-22 18:56:57.100113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2019-11-22 18:56:58.357108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2019-11-22 18:56:58.432503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2019-11-22 18:56:58.432679: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-11-22 18:56:58.433099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-11-22 18:56:58.433463: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-11-22 18:56:58.435936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2019-11-22 18:56:58.436136: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-11-22 18:56:58.436559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-11-22 18:56:59.016772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-22 18:56:59.016930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2019-11-22 18:56:59.017022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2019-11-22 18:56:59.017714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6273 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-11-22 18:56:59.409303: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: No unary variant device copy function found for direction: 1 and Variant type_index: class tensorflow::data::`anonymous namespace'::DatasetVariantWrapper
	 [[{{node while_input_4/_12}}]]
	 [[Func/while/body/_1/input/_47/_20]]
2019-11-22 18:56:59.409640: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: No unary variant device copy function found for direction: 1 and Variant type_index: class tensorflow::data::`anonymous namespace'::DatasetVariantWrapper
	 [[{{node while_input_4/_12}}]]
Traceback (most recent call last):
  File "D:/work/python/PycharmProjects/tftest/test.py", line 83, in <module>
    f()
  File "D:\work\python\PycharmProjects\tftest\venv-gpu\lib\site-packages\tensorflow_core\python\eager\def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "D:\work\python\PycharmProjects\tftest\venv-gpu\lib\site-packages\tensorflow_core\python\eager\def_function.py", line 526, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "D:\work\python\PycharmProjects\tftest\venv-gpu\lib\site-packages\tensorflow_core\python\eager\function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "D:\work\python\PycharmProjects\tftest\venv-gpu\lib\site-packages\tensorflow_core\python\eager\function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "D:\work\python\PycharmProjects\tftest\venv-gpu\lib\site-packages\tensorflow_core\python\eager\function.py", line 511, in call
    ctx=ctx)
  File "D:\work\python\PycharmProjects\tftest\venv-gpu\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal:  No unary variant device copy function found for direction: 1 and Variant type_index: class tensorflow::data::`anonymous namespace'::DatasetVariantWrapper
	 [[{{node while_input_4/_12}}]]
	 [[Func/while/body/_1/input/_47/_20]]
  (1) Internal:  No unary variant device copy function found for direction: 1 and Variant type_index: class tensorflow::data::`anonymous namespace'::DatasetVariantWrapper
	 [[{{node while_input_4/_12}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_f_61]

Function call stack:
f -> f


Process finished with exit code 1
tensorflow/tensorflow

Answer questions jsimsa

The problem here is in an assumption that the placer algorithm makes -- that any tensor can be copied between different devices -- which is generally not true (e.g. for dataset variant tensors). The proper way to fix this would be to make sure that placer only allows placement that collocate a node with its non-copyable inputs.

useful!

Related questions

ModuleNotFoundError: No module named 'tensorflow.contrib'
Error occurred when finalizing GeneratorDataset iterator
ModuleNotFoundError: No module named 'tensorflow.contrib'
When importing TensorFlow, error loading Hadoop hot 4
The flag 'log_dir' is defined twice. hot 3
[TF 2.0] Dataset has no attribute 'make_one_shot_iterator' hot 3
Lossy conversion from float32 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning. hot 3
TF2.0 AutoGraph issue hot 3
Error loading tensorflow hot 3
AttributeError: module 'tensorflow' has no attribute 'set_random_seed' hot 3
AttributeError: module &#39;tensorflow&#39; has no attribute &#39;Session&#39; hot 3
No tf.lite.experimental.nn.bidirectional_dynamic_rnn ops is finded hot 3
AttributeError: module 'tensorflow' has no attribute 'app' hot 3
Incorrect Error TypeError: padded_batch() missing 1 required positional argument: &#39;padded_shapes&#39; hot 3
tensorflow2.0 detected 'xla_gpu' , but 'gpu' expected hot 2
Github User Rank List