CODEWITHSUNDEEP
daskdask-distributeddask-delayed
Patrick Mineault
Patrick Mineault

Reputation: 741

Long running workers blocking GIL timeout errors

I'm using dask-distributed with a local setup (LocalCluster with 5 workers) on a dask.delayed workload. Most of the work is done by the vtk Python bindings. Since vtk is C++ based I think that means the workers don't release the GIL when in a long-running statement. When I run the workload, my terminal prints out a bunch of errors like this:

Traceback (most recent call last):
  File "C:\Users\patri\AppData\Local\Continuum\anaconda3\lib\site-packages\distributed\comm\core.py", line 221, in connect
    _raise(error)
  File "C:\Users\patri\AppData\Local\Continuum\anaconda3\lib\site-packages\distributed\comm\core.py", line 204, in _raise
    raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://127.0.0.1:49721' after 10 s: connect() didn't finish in time

My workload continues fine however - I get a bunch of errors on the command line but it keeps chugging along. So I think the workers aren't crashing, but the heartbeat communication stops. Since I don't want to mess with vtk internals to release the GIL, how can I fix the errors? I get so many of these benign timeout errors that I can't see any real errors that might happen.

Upvotes: 1

Views: 417

Answers (1)

Victor Ruiz
Victor Ruiz

Reputation: 1252

Release the GIL temporally by sleeping the VTK event loop thread. If you are using a vtkWindowRendererInteractor instance, create a timer with a callback which sleeps the execution a bit using the sleep builtin.

Upvotes: 1

Related Questions