@FreddyLab thank you for bringing that to our attention. I'll submit a fix for the documentation. Glad your issue is resolved now!

I haven't tested with TF 2.2 or tf-nightly yet, because I haven't found any links to download these Libtensorflow JNI packages versions. At the URL https://storage.googleapis.com/libtensorflow-nightly I only found libs for Linux and MacOs but none for Windows. Do you know where I can download TF 2.2 or nightly libtensorflow versions for Windows?

That's odd I don't know why you cannot see it in the browser window, but they are there. Might be a listing issue. gsutil might list them better. Either way, here are the ones from 09/17. CPU CPU JNI GPU GPU JNI

Glad we were able to solve the first issue, sorry about the new issues you are facing. Do you mind uploading some logs so I can see if that's something we can fix on our end? The package was built with our CUDA 11 toolchain.

So that link is a browser representation of the GCS bucket libtensorflow-nightly. I may be describing it badly, but regardless I think the file you're looking for is this which was built last night.

Sorry I forgot to update this thread, but the latest GCS builds are built with the following computes:

sm_35,sm_50,sm_60,sm_70,sm_75,compute_80

Fix https://github.com/tensorflow/custom-op/issues/74 Fix https://github.com/tensorflow/custom-op/issues/78

I'm trying to build the example files on Windows, but I keep getting the same error after following the tutorial several times. Bash keeps telling me the following:

Code snippet:

$ bazel build tensorflow_zero_out:python/ops/_zero_out_ops.so Loading: Loading: 0 packages loaded Analyzing: target //tensorflow_zero_out:python/ops/_zero_out_ops.so (0 packages loaded, 0 targets configured) INFO: Analyzed target //tensorflow_zero_out:python/ops/_zero_out_ops.so (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 5] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: C:/users/patrik.veges/_bazel_patrik.veges/gd5nvtb3/external/local_config_tf/BUILD:17:8: Executing genrule @local_config_tf//:tf_header_include failed (Exit 35584) 0 [] bash 465 cygwin_exception::open_stackdumpfile: Dumping stack trace to bash.exe.stackdump Target //tensorflow_zero_out:python/ops/_zero_out_ops.so failed to build Use --verbose_failures to see the command lines of failed build steps. ERROR: C:/users/patrik.veges/_bazel_patrik.veges/gd5nvtb3/external/local_config_tf/BUILD:3:11 Executing genrule @local_config_tf//:tf_header_include failed (Exit 35584) INFO: Elapsed time: 1.299s, Critical Path: 1.19s INFO: 0 processes. FAILED: Build did NOT complete successfully FAILED: Build did NOT complete successfully

I tried docker pull tensorflow/tensorflow:custom-op-gpu-ubuntu16 and run it. Then tried './configure.sh' However, it detected that no tensorflow-gpu is installed in this container, and has to uninstall the already installed tensorflow and install atensorflow-gpu version. Shouldn't your docker image already has tensorflow-gpu installed?

Taking a look now, and will triage it to bazel. I second @gunan, thank you @Flamefire for bisecting. @Flamefire I think opening a PR is not a bad idea, I don't know if we will decide to merge it though.

@gunan unfortunately we only have the C API uploaded to GCS via our libtensorflow distributable. Libtensorflow_cc is being planned for Q4 of this year.

Hello Pratik,

As we discussed on our email correspondences, I was encouraging you to create a bazel test within tensorflow. Regardless, I will still take ownership of the miscommunication. As a courtesy since I did originally tell you that a bazel target would easily allow me to execute, I tried running your bazel target and ran into the following issue main/python_cpp.cc:2:20: fatal error: Python.h: No such file or directory. Additionally I think you have some typos in the setup instructions, particularly the copy instructions.

Unfortunately, I am unable to help you further, but hopefully you'll be able to resolve the issue. I recommend pursuing the avenue of trying to find a way to print the error message we discussed in the previous comments. Good luck!

Unfortunately it appears my change did not work, digging into why today.

Hello @motrek I'll trigger a new push to GCS. Hopefully this next one or by tomorrow's version we can have something working.

https://github.com/tensorflow/tensorflow/commit/049fa9d21b28dbbb6780fd09852cde2f1a09b352 should fix this. I'll keep monitoring to see if there are any other issues.

@angerson I'll send a fix internally that matches the pip package config.

Yes, bazel and makefile serve a similar purpose, but bazel targets are significantly easier to run for us.

Unfortunately, I have not tried to reproduce your environment due to resource and policy reasons, but hopefully maybe someone else in the community has seen this before.

Sorry, but I personally have not had cycles to look into libtensorflow_cc. I'll see if I can carve time in Q4.

Bazel is just the build system, so you'll need toolchains and python dependencies installed and appropriate environment variables set. Since you're just trying to import a pip package, I don't think bazel is an issue.

Thanks for your patience and answering the questions I had. Unfortunately, I'm not sure what else the issue could be. I think the fact that a simple pybind11 module can be loaded shows that the issue is in TensorFlow, but without a simpler unit test or bazel target, I cannot reproduce this easily.

I'll keep this issue open in the event someone else has seen similar errors.

Oh there is no need for it to be with bazel. I thought it may be convenient for you. Does it work without bazel?

Update the README with new docker container links for TF >= 2.3.0

The fact that it is happening or basic imports may mean that it's a systemic issue where the sandwich of your C++ code -> Python C API -> Python Import of TensorFlow -> Using pybind11 to call C++ is just something that is not supported. I think the next step I'd try is to setup a barebones pybind11 Python module and see if you can import it via your code. So for example I have a math pybind11 + bazel module here you can try. Then we can at least narrow it to either TensorFlow or pybind11.

Update the README with new docker container links for TF >= 2.3.0

Unfortunately, I cannot help you without a bazel target. Due to resource restrictions, I cannot try and execute client source code and repro their environment for user specific issues. Can you please convert this into a bazel target for me to run? That way I can just easily execute and repro the error and see if it occurs on all operating systems. Unfortunately, I will not be able to duplicate your setup and run your individual source code.

From what I can see, this is not a standalone issue with TensorFlow, but rather an issue that propagates when importing the TensorFlow Python module from C++. It may be better suited for StackOverFlow as well as per our guidelines.

I also noticed you mentioned that the address of the error is an unreadable character. Have you tried a try catch block and then reading the Python error that has been set using PyErr_PrintEx (docs)?

Yes, sorry about that, my intent was to post the pybind11 docs link that you provided. Regardless, it appears you've already tried that.

Can you potentially try simpler imports? That might help us narrow down if it's python or pybind11 calling back into pybind11.

from tensorflow.python import _pywrap_utils for example.

If you can provide me with a branch to checkout with a bazel target that is the bare minimum py::module::import and instructions to repro, I can see if I can repro the issue on a Windows VM.

Can you try importing the module using pybind11 import functionality as documented here?

Okay so I guess you're trying to invoke a Python file from C++. And the regular import works, so there is something happening from the C++ invocation. Do you have exceptions enabled? Can you potentially print the error in your C++ code before the line that throws the error is executed?

Unfortunately I'm not sure what we can do here.

So a simply import tensorflow in Python3.8 is failing on Windows? Even if you remove your code? Usually throwing error_already_set from C++ leads to a Python exception. Can you somehow see the Python exception behind that C++ exception?

Can you please provide more information? The full stacktrace of the error perhaps. We did start using pybind11 in version 2.1 so this is probably related.

Our pybind11 code has a lot of error checks that throw that particular error. Example

Unfortunately since the default import for tensorflow is working, I don't think the issue lies in the tensorflow import. There is perhaps something in your code that is throwing one of the pybind11 errors.

We used the same Python3.7 auditwheel across all binaries.

Actually sorry, my commit deleting the SWIG stuff went in early. If you'd like you can make the PR to the 2.2 branch and others can cherrypick that commit.

That's a good idea, I'll look into this now. Thanks!

https://github.com/tensorflow/tensorflow/commit/7fd9e941e10f76abcea3a398fd02eb93ca343012#diff-38c70d53f71218cab252a7f458600d6d https://github.com/tensorflow/tensorflow/commit/5fb9558424c2bf8f8c3de762a34bec59c11fa26e#diff-38c70d53f71218cab252a7f458600d6d

2.3 will have these changes.

Okay, sounds good. I'll approve this so anyone can cherrypick easily into any branch. I'll send a change to remove that rule as I don't think anyone still uses it at HEAD.

What version or branch of TensorFlow is this for? I got rid of all the SWIG compile logic around 2.3.

Unfortunately, I'm not sure, maybe @rmlarsen may have more insight.

https://github.com/tensorflow/tensorflow#continuous-build-status

Thank you @jsimsa

So just to clarify, you're moving the dependency to if_static, but what if the user accesses the API from open source? What is the expected behavior?

Are the failures related? Might be broken at HEAD. I'll force a rerun.

So I had reverted my config change on 07/14, so if you want to be safe it may be worth pinning to 07/15 or trying this custom whl (it was not uploaded to pypi) I've created with my config change and fix to the cuda issue. I've resubmitted my config change today after testing.

I was able to repro and can confirm the new config fixes the issue.

@tgaddair yes my machine had CUDA available. Is there any way it would be possible for you to test a new tf-nightly image in your environment from this link?

I've renovated the release config but haven't published it out of fear that this bug still exists. This is a Python3.6 version, but if you can confirm installing this whl works for you I can push the updates to pypi.

We normally have a smoke test that is run before we upload. For some reason I'm unable to repro this breakage:

$ python --version
Python 3.7.7
(venv37) $ python
Python 3.7.7 (default, Mar 10 2020, 17:25:08) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'2.4.0-dev20200714'
>>> from tensorflow.python._pywrap_tensorflow_internal import *
>>>

If you have some repro steps please let me know.

Sorry for the breakage. I've deleted 07/14 builds across all packages and will roll back the config changes.

@shwetaoj no problem! Glad I could help!

This may not be TensorFlow specific but perhaps related to your installation of Python 3.5 on your machine. I think you need to add the location of python3.5m/pyconfig.h to your path, or symlink it to that location?

https://github.com/okfn/piati/issues/65 https://github.com/BVLC/caffe/issues/410

^ Similar issues that were solved that way. Have you been able to build TF prior to in the same env?

Are you a member of SIG Build? This might be a better conversation to have there. I vaguely remember some discussions about ARM CI, although I myself do not have plans to implement it.

2.3 will have libtensorflow. Additionally we have nightly packages on GCS. You can find the links on the main README.

Just to clarify we have only resurrected the libtensorflow (C API) build. I have not looked into libtensorflow_cc. I'm not sure what the build issues there are.

If you need the cherrypick them, the list is tagged on the internal bug.

Sorry I'm not sure, I have not seen this error before. @mihaimaruseac any chance you have?