CODEWITHSUNDEEP
pythondockerdaskdask-distributed
ps0604
ps0604

Reputation: 1071

Reload Dask worker containers automatically on code change

I have the Dask code below that submits N workers, where each worker is implemented in a Docker container:

default_sums = client.map(process_asset_defaults, build_worker_args(req, numWorkers))
future_total_sum = client.submit(sum, default_sums)
total_defaults_sum = future_total_sum.result()

where process_asset_defaults is a method in a worker.

The problem is that in a development environment when I change the worker's code I need to restart all the containers manually for the change to take effect.

Is there a way to reload the worker with the new code without restarting the workers?

Note: the code resides in a Docker volume, I change it directly in the volume with Visual Studio Code.

Upvotes: 6

Views: 400

Answers (2)

SultanOrazbayev
SultanOrazbayev

Reputation: 16551

If the code that changes is the content of the functions, then it might be possible to use autoreload:

import importlib
importlib.reload(process_asset_defaults) # if this is the function that needs updating

See blog or docs for some further details.

Upvotes: 1

Robiot
Robiot

Reputation: 98

You could make make a variable with the current content of the file, then make a loop which checks if the file content is not equals to the one before, and if it is not you can do stuff

before = open("/code/app/worker.py").read()

while True:
   current = open("/code/app/worker.py").read()
   if current != before:
       dostuff()
       before = current

Upvotes: 1

Related Questions