CODEWITHSUNDEEP
daskdask-distributed
tjcrone
tjcrone

Reputation: 21

Using NFS with Dask workers

I have been experimenting with using an NFS shared drive with my user and Dask workers. Is this something that can work? I noticed that Dask created two files in my home directory, global.lock and purge.lock, and did not clean them up when workers were finished. What do these files do?

Upvotes: 1

Views: 386

Answers (2)

MRocklin
MRocklin

Reputation: 57311

It is entirely normal to use the NFS to host a user's software environment. The files you're seeing are used by a different system altogether.

When Dask workers run out of space they spill excess data to disk. An NFS can work here, but it's much nicer to use local disk if available. This is usually configurable with the --local-directory dask-worker keyword, or the temporary-directory configuration value.

You can read more about storage issues with NFS and more guidelines here: https://docs.dask.org/en/latest/setup/hpc.html

Upvotes: 1

mdurant
mdurant

Reputation: 28684

Yes, Dask can be used with an NFS mound, and indeed you can share configuration/scheduler state between the various processes. Each worker process will use its own temporary storage area. The lock files are safe to ignore, and their existence will depend on exactly the workload you are doing.

Upvotes: 0

Related Questions