CODEWITHSUNDEEP
python-3.xdaskslurmdask-distributeddask-jobqueue
michaelgbj
michaelgbj

Reputation: 294

Worker log file gets mangled when log_directory is set

I used to have the worker logs as such:

./slurm-<id>.out
...

So I wanted to have SLURMCluster writes the worker logs in a separate directory (as opposed to the current working dir), so I provided "log_directory" as an input argument as such.

from dask_jobqueue import SLURMCluster
c = SLURMCluster(queue_name, log_directory="dirA")
c.scale_up(1)

And the logs end up like this:

./dirA/dask-worker-<id>.out
./dirA/dask-worker-<id>.err

And all the worker logs are piped to the .err file while the .out file is empty.

What am I missing in the SLURMCluster configuration to set it correctly, such that the .out should contain the logs and the .err should contain worker errors.

Upvotes: 0

Views: 108

Answers (0)

Related Questions