CODEWITHSUNDEEP
pythontensorflowsupervisord
tubadc
tubadc

Reputation: 772

Supervisor can't load Cuda libs , PATH IS NOT MISSING, works without supervisor

I've a python application with Flask, tensorflow-gpu ..., that runs without a problem when I run:

gunicorn server:app -b localhost:8000

but when I run it with the supervisor, it gives me the error

ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

I'm not using virtualenv.

here is my supervisor config

[program:appserver]
command = gunicorn server:app -b localhost:8000
directory = /storage/appserver
user = root
stdout_logfile = /home/deploy/appserver/logs/gunicorn/gunicorn_stdout.log
stderr_logfile = /home/deploy/appserver/logs/gunicorn/gunicorn_stderr.log
redirect_stderr = True

What am I doing wrong?

If the app runs directly at the command line, why supervisor can't run it?

I've the env vars at the bashrc, like I said, it works if I run from the command line...

export PATH=/usr/local/cuda-9.0/bin/${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64/


root@xxxxx:/home/xxxxx# echo $PATH
/usr/local/cuda-9.0/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
root@xxxxxx:/home/xxxxx# echo $LD_LIBRARY_PATH
/usr/local/cuda-9.0/lib64/

Upvotes: 2

Views: 987

Answers (1)

estevo
estevo

Reputation: 1011

Supervisor don't load environment variables, it doesn't depend on bash, so you must load environment variables in supervisor configuration directly.

Here my configuration as example:

[program:hoot_api_ml]

user           = hoot
environment    = LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
directory      = /home/hoot/backend/hoot/ml
command        = gunicorn3 -c ../../conf/api_ml/gunicorn.py api:APP

autostart      = true
autorestart    = true

stderr_logfile = /var/log/hoot/api_ml_supervisor.err
stdout_logfile = /var/log/hoot/api_ml_supervisor.log

stopsignal     = INT

Upvotes: 1

Related Questions