Can't make TensorFlow 2.4.1 (CPP) compile on Windows

Question

I am trying to build TensorFlow 2.4.1 C++ API on Windows 10 and I am having issues.

What I've done so far:

1. Download TensorFlow Source from the official repo https://github.com/tensorflow/tensorflow and switched to the official v2.4.1 tag
2. Download and install Python 3.6.8 x64
3. Created a virtual env with python 3.6.8
4. Created and installed requirements.txt based on the data here
5. Download and install CUDA 11.0 and cuDNN v8.0.4.30 (for CUDA 11)
6. Downloaded and installed msys2 and set it's location in PATH
7. Download and install bazel (3.1.0) as it is the most recent entry here

Then I run the configuration process with python configure.py I configure for C++ build (tensorflow_cc) with GPU support Here I had some problem that apparently during the config process windows-style backslashes are accepted as a valid input, but then when you actually run bazel compilation they cause problems, so I reran my configuration to provide linux-style backslashes. Thus CUDA and cuDNN were successfully detected and compilation started.

The full contents of my .tf_configure.bazelrc are below

build --action_env PYTHON_BIN_PATH="D:/code/sdk/tensorflow/venv/Scripts/python.exe"
build --action_env PYTHON_LIB_PATH="D:/code/sdk/tensorflow/venv/lib/site-packages"
build --python_path="D:/code/sdk/tensorflow/venv/Scripts/python.exe"
build --config=xla
build --action_env TF_CUDA_VERSION="11.0"
build --action_env TF_CUDNN_VERSION="8.0.4"
build --action_env TF_CUDA_PATHS="C:/Program Files/NVIDIA GPU Computing 
Toolkit/CUDA/v11.0,D:/code/sdk/cudnn-11.0-windows-x64-v8.0.4.30/cuda"
build --action_env CUDA_TOOLKIT_PATH="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0"
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="5.0"
build --config=cuda
build:opt --copt=/arch:AVX
build:opt --host_copt=/arch:AVX
build:opt --define with_default_optimizations=true
build --define=override_eigen_strong_inline=true
test --flaky_test_attempts=3
test --test_size_filters=small,medium
test:v1 --test_tag_filters=-benchmark-test,-no_oss,-no_windows,-no_windows_gpu,-no_gpu,-oss_serial
test:v1 --build_tag_filters=-benchmark-test,-no_oss,-no_windows,-no_windows_gpu,-no_gpu
test:v2 --test_tag_filters=-benchmark-test,-no_oss,-no_windows,-no_windows_gpu,-no_gpu,-oss_serial,- 
v1only
test:v2 --build_tag_filters=-benchmark-test,-no_oss,-no_windows,-no_windows_gpu,-no_gpu,-v1only
build --action_env TF_CONFIGURE_IOS="0"

About 20 minutes into the compilation however it failed with the following error:

ERROR: D:/code/sdk/tensorflow/tensorflow/stream_executor/cuda/BUILD:366:1: C++ compilation of rule '//tensorflow/stream_executor/cuda:cudnn_stub' failed (Exit 2): python.exe failed: error executing command

And the reason the command fails to execute is

bazel-out/x64_windows-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cudnn_header\third_party/gpus/cudnn/cudnn.h(61): fatal error C1083: Cannot open include file: 'cudnn_ops_infer.h': No such file or directory

And here I am wondering what is going on? I already provided a path to cuDNN, but apparently bazel doesn't really know about it, even though it previously acknowledged that the path I have provided is correct. Am I missing some environment variable that I need to set to instruct where cuDNN is? Has anyone built TF C++ v2.4.1 on Windows? There is so little information online, even the official page says nothing about Windows builds. It's only Linux and Mac...

Answer 1

As I was running out of ideas I decided to go take a look at the Bazel build scripts for CUDA

In <REPO>\tensorflow\third_party\gpus\cuda_configure.bzl I saw that cuDNN path is read from env variable CUDNN_INSTALL_PATH and if not present it will default to /usr/local/include?

Anyway, tried set CUDNN_INSTALL_PATH=D:/code/sdk/cudnn-11.0-windows-x64-v8.0.4.30/cuda and WOOHOO It compiled!

(Pro Tip: Set the env var without any quotes and with linux-style slashes...)