@silverark How are you installing protoc? It looks like you're missing the default protos from your installation. Typically they're in /usr/local/include after installation, like so:

jhseu0:~$ locate any.proto
/usr/include/google/protobuf/any.proto

This is actually fixed at master following the instructions at: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.md

The fix didn't make it into TensorFlow 2.3. (The go get -d command will warn about package not found, but you can ignore it since it's built with go generate)

Note that you must build an updated version of libtensorflow.so as indicated in the instructions, otherwise you'll get errors.

Spark 3.0.0 was officially released in June. The project is providing Scala 2.12 binaries.

I think the Tensorflow connector should now be targeting this newer version of Spark by default, and perhaps Spark 2.x in a branch. Spark 2.4.7 and other maintenance releases are likely in future, but I expect they will be bug fixes and backports or new features.

Can you verify whether this is still an issue in recent versions of Hadoop before we merge? If it still is an issue, I don't mind merging this.

Can this continue to use ${scala.binary.version}?

This is needed after https://github.com/tensorflow/tensorflow/commit/b129d3dc1cfc1944b874c514724146df971ea935

To make it more idiomatic and easier to install. See #35133

Gunhan, any objections to doing this? It adds 1.3 MB of generated files to the repo.

We had a discussion with the Go team who discouraged checking this in because other projects had faced issues with conflicts between generated protobufs and the protobuf files generated by bazel. Closing for now.

Users should run go generate as mentioned in the README.

@JacobSMoller Yeah, go get doesn't work for now. I made a pull request to fix it, but there were some objections to supporting both the bazel and generated protobufs to make it work.

For that error: it occurs when there's a version mismatch between libtensorflow.so and the TF ops in the source code. See my command above for building libtensorflow.so.

System information

• OS Platform and Distribution (Macos catalina 10.15.4):
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow installed from (source or binary):
• TensorFlow version:
• Python version:
• Installed using virtualenv? pip? conda?:
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:
• GPU model and memory:

Describe the problem

Provide the exact sequence of commands / steps that you executed before running into the problem

Following https://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.mdhttps://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.md

go generate github.com/tensorflow/tensorflow/tensorflow/go/op fails with failed to process "api_def_VarHandleOp.pbtxt": Attribute allowed_devices not defined in base api for VarHandleOp exit status 1 go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/generate.go:18: running "go": exit status 1

go test github.com/tensorflow/tensorflow/tensorflow/go passes

building or running go code using tensorflow then proceeds to fail with the following output go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core latest go: finding github.com/tensorflow/tensorflow/tensorflow/go latest go: finding github.com/tensorflow/tensorflow/tensorflow latest build command-line-arguments: cannot load github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto: cannot find module providing package github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto

the code run is the example that prints tf library version from https://www.tensorflow.org/install/lang_go

Any other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Fixed as part of: https://github.com/tensorflow/tensorflow/commit/e5e495db7bee77cd0fd5dda3b06bd743cbcf1ef8

Note that it requires from master: bazel build -c opt //tensorflow/tools/lib_package:libtensorflow tar -xzvf the package in /usr/local go generate github.com/tensorflow/tensorflow/tensorflow/go/op

We don't currently support loading local files this way, although it'll work at some point in the near future. For now, you can load them outside of tf.data as numpy arrays and feed them instead.

master was renamed to dispatcher in https://github.com/tensorflow/tensorflow/commit/c3f2d3d5710caa789eb54a4d24280a818b9375ba

Fixes

#39744

Summary

After running the following in accordance with the installation instructions for go:

$ go get -d github.com/tensorflow/tensorflow/tensorflow/go
$ go generate github.com/tensorflow/tensorflow/tensorflow/go/op
$ go test github.com/tensorflow/tensorflow/tensorflow/go

The first command generates error message unrelated to this PR, a fix for which is ready to pull. That error message is relatively low impact and should not impact the functionality of the library after the go generate command is run. However, the third command generates the following error:

$ go test github.com/tensorflow/tensorflow/tensorflow/go
../../../go/src/github.com/tensorflow/tensorflow/tensorflow/go/saved_model.go:25:2: cannot find package "github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto" in any of:
	/home/abls/go/src/github.com/tensorflow/tensorflow/tensorflow/go/vendor/github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto (vendor tree)
	/usr/local/go/src/github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto (from $GOROOT)
	/home/abls/go/src/github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto (from $GOPATH)

This is caused by a bad import that seemed to be from an accident during a refactor. The problematic import that occurs in several files is:

import corepb "github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto"

This line is supposed to point to the generated files, however the generated files are actually placed in github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto due to the following line in several proto files in tensorflow/core/protobuf read by protoc during generation:

option go_package = "github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto";

The fix is simply to correct the import lines to point to the actual location of the files.

Needed to change this a bit to match the BUILD rule, which is required for bazel compatibility: https://github.com/tensorflow/tensorflow/commit/e5e495db7bee77cd0fd5dda3b06bd743cbcf1ef8

MacOS version(10.15.4)

1. I followed the instruction of https://www.tensorflow.org/install/lang_c ,I installed the c Library .but it doesn't word.the error is > hello_tf.c:2:10: fatal error: 'tensorflow/c/c_api.h' file not found

2. github.com/tensorflow/tensorflow/tensorflow/go/saved_model.go:25:2: cannot find package "github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto" in any of: /usr/local/go/src/github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto (from $GOROOT) /Users/moses/workspace/godemo/src/github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto (from $GOPATH)

how to solve this problem

Fixed with https://github.com/tensorflow/tensorflow/commit/e5e495db7bee77cd0fd5dda3b06bd743cbcf1ef8

This PR fixes the data downloading issue in the example code.

Reproduce: On a cluster with multiple GPUs per worker node, with spark.resources.tasks.gpu.amount set to 1, running the original example will trigger an error related to data downloading.

Cause: There will be multiple tasks running on the same worker and each task will try to write the data to the same path, which will corrupt the data.

Fix: Randomize the file path.