CODEWITHSUNDEEP
daskdask-distributeddask-delayed
TheCodeCache
TheCodeCache

Reputation: 962

How do I capture dask-worker console logs in a file?

In the below, I want to capture "dask_client_log_msg" and other task-logs in one file and "dask_worker_log_msg" and other client-logs in a separate file. As obviously client will run in a separate process altogether than the worker. So I need one process should log all its message in a separate file. Thanks!

def my_task():
  print("dask_worker_log_msg")
  ...

client = Client(<scheduler_address>)

future = client.submit(my_task)
print("dask_client_log_msg")
...

Upvotes: 5

Views: 6993

Answers (3)

mateuszb
mateuszb

Reputation: 1102

Structured logs can help here, example from the documentation:

def myfunc(x):
    start = time()
    ...
    stop = time()
    dask.distributed.get_worker().log_event("runtimes", {"start": start, "stop": stop})

futures = client.map(myfunc, range(10))
client.get_events("runtimes")

Upvotes: 0

Michał Zawadzki
Michał Zawadzki

Reputation: 752

Here's a solution if you're trying to implement a Dask cluster and need the logs from all jobs that it runs (including logs from your scripts from print or logger.info):

  1. Add a redirect in your bash script starting the worker: dask-worker >> dask_worker.log 2>&1
  2. In your script, set your logger to dask.distributed, like so: logger = logging.getLogger("distributed.worker")
  3. Configure the log format in .config/dask/distributed.yaml

See also How to capture logs from workers from a Dask-Yarn job?

Upvotes: 3

MRocklin
MRocklin

Reputation: 57251

You can get logs from your workers with the Client.get_worker_logs method. You can also download logs from the dashboard in the info pane.

Upvotes: 3

Related Questions