CODEWITHSUNDEEP
python-3.xazurescikit-learnazure-machine-learning-serviceazureml-python-sdk
mark_r
mark_r

Reputation: 1

Azure ML Pipeline fails while running a grid search CV on a cluster

I implemented a gridsearchcv on Azure ML as a pipeline but I keep getting an error that says "User program failed with TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGKILL(-9)}"

I tried changing the package versions but could not get it to work. The code runs well without error when I run it as a script but fails when I run it as a pipeline.

Any idea on how to fix this error?

Upvotes: 0

Views: 967

Answers (1)

Ecstasy
Ecstasy

Reputation: 1864

This error ... excessive memory usage causing the Operating System to kill the worker The exit codes of the workers are {SIGKILL(-9)}.

Thank you will kinsman and pnmartinez. Posting your suggestion as an answer to help other community members.

This is caused by the n_jobs = -1 which means parallelization in all CPUs (until last element of the array of cpus, hence -1).

You can try removing n_jobs = -1 in GridSearchCV and put it inside classifier or try n_jobs = 1 to deactivate the parallelization.

You can refer to Memory leak using gridsearchcv and TerminatedWorkerError

Upvotes: 1

Related Questions