CODEWITHSUNDEEP
pythononnxruntime
Oguz Hanoglu
Oguz Hanoglu

Reputation: 341

Why does onnxruntime fail to create CUDAExecutionProvider in Linux(Ubuntu 20)?

import onnxruntime as rt
ort_session = rt.InferenceSession(
            "my_model.onnx",
            providers=["CUDAExecutionProvider"],
)

onnxruntime (onnxruntime-gpu 1.13.1) works (in Jupyter VsCode env - Python 3.8.15) well when providers is ["CPUExecutionProvider"]. But for ["CUDAExecutionProvider"] it sometimes(not always) throws an error as:

[W:onnxruntime:Default, onnxruntime_pybind_state.cc:578 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

I tried following the provided link in the error, and tried different setups in the conda environment to test the code with various version combinations.

Upvotes: 5

Views: 20912

Answers (5)

Mitch McMabers
Mitch McMabers

Reputation: 4550

The problem is that ONNX doesn't know how to search for CUDA. PyTorch knows how to search for it, and adds it to Python's internal path, so that ONNX can later find it.

The bug/issue is with ONNX library. There is a great workaround here:

https://github.com/cubiq/ComfyUI_IPAdapter_plus/issues/238

Upvotes: 3

Louis Lac
Louis Lac

Reputation: 6426

I had the same issue but with TensorRT TensorrtExecutionProvider:

[W:onnxruntime:Default, onnxruntime_pybind_state.cc:614 CreateExecutionProviderInstance] Failed to create TensorrtExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#requirements to ensure all dependencies are met.

Look like in this case one have to import tensorrt and have to do so before importing onnxruntime (GPU):

import tensorrt
import onnxruntime as ort

# Inference with ONNX TensorRT

Upvotes: 0

thawro
thawro

Reputation: 136

In my case the following helped:

  1. uninstall onnxruntime
  2. uninstall onnxruntime-gpu
  3. install optimum[onnxruntime-gpu]

more here

Upvotes: 5

peng yao
peng yao

Reputation: 11

  1. You should check your onnxruntime-gpu version ,cuda version and cudnn version in this link 'https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html'
  2. Use 'nvcc -V' and 'cat /usr/local/cuda/include/cudnn_version.h' make sure your env is useful.
  3. 'import torch' is necessary,the code as is follows:
  • import torch
  • import onnxruntime as rt
  • providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
  • session = rt.InferenceSession("yolo.onnx", providers=providers)

Upvotes: 1

Oguz Hanoglu
Oguz Hanoglu

Reputation: 341

Replacing:

import onnxruntime as rt

with

import torch
import onnxruntime as rt

somehow perfectly solved my problem.

Upvotes: 13

Related Questions