CODEWITHSUNDEEP
daskdask-distributed
MakleBirt
MakleBirt

Reputation: 591

Where is the correct location for Dask Worker configuration file and the Dask Scheduler configuration file?

I am attempting to find the correct location of Dask configuration files. I have a number of questions related to configuring Dask.

$ dask-worker --version
dask-worker, version 2.3.2

Do the Dask Worker and Dask Scheduler share the same configuration file or do they use different configuration files?

I am unclear if there are configuration variables that are specific to Dask Worker and Dask Scheduler. Is there a list of the valid configuration variables for Dask Worker and Dask Scheduler?

Where are the correct locations of the Dask Worker and Dask Scheduler configuration files?

I have found three different configuration files across my system and the Dask documentation:

  1. ~/.config/dask/distributed.yaml
  2. ~/.config/dask/dask.yaml
  3. ~/.dask/config.yaml

On my Dask Worker and Dask Scheduler machines, I find a file located at ~/.config/dask/dask.yaml which does not contain much information. I am not sure what should go into this file or if/where it is ever called by the Dask library.

I also see a file at ~/.config/dask/distributed.yaml that contains much more information. This looks more like the configuration I was expecting. I can see that these configuration are also loaded by Dask in distributed/config.py

A third file (~/.dask/config.yaml) makes an appearance in the documentation. To quote the documentation:

Dask accepts some configuration options in a configuration file, which by default is a .dask/config.yaml file located in your home directory.

I do not see this file on my system. Am I responsible for creating this configuration file? I never see this file referenced in the repository. Why does the documentation differ from the source code?

Can I print a list of all active configuration variables for both the Worker and the Scheduler?

Is there a way, either on the command line or in Python, where I can inspect the active configurations?

Upvotes: 0

Views: 2101

Answers (1)

MRocklin
MRocklin

Reputation: 57281

For documentation on Dask's configuation system please see https://docs.dask.org/en/latest/configuration.html

That page says:

Configuration is specified in one of the following ways:

  • YAML files in ~/.config/dask/ or /etc/dask/
  • Environment variables like DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING=True
  • Default settings within sub-libraries

I've removed the page that you were looking at in this PR: https://github.com/dask/distributed/pull/3038

Upvotes: 1

Related Questions