CODEWITHSUNDEEP
pythonmacoscelerycelery-task
Daniel Stephens
Daniel Stephens

Reputation: 3209

Process started by Celery does not stop when main process is terminated

I have a celery worker that spawns a new process for each incoming task. 

@app.task
def add(a, b):
    p = subprocess.Popen(args=['add', a, b], stdout=PIPE, stderr=PIPE)
    stdout, stderr = p.communicate()

When the celery process gets terminated by SIGKILL or a cold shutdown, the Python process it started continues to run. Even in the process hierarchy macOS does say the parent process does not exist anymore. Isn't a child process supposed to be killed as well when the parent process gets unexpectedly terminated?

enter image description here

Upvotes: 1

Views: 1756

Answers (1)

Edeki Okoh
Edeki Okoh

Reputation: 1834

The simple answer is no, a child process will not automatically be killed if you kill the parent process.

This github question goes into a bit more detail, but the tldr is that Celery is designed to not kill the child process accidentally if the parent is killed in case the child process needs to finish after the parent has already been executed. For example a task that tells other tasks to update various databases. After the parent is called you can open up the parent worker, but you do not necessarily want the child processes to be killed if you stop the parent worker.

I use this function to directly kill child processes:

from psutil import Process

def terminate_process(pid):

  process = Process(pid)
  for child in process.children(recursive=True):
    child.kill()
  process.kill()

Which you can call to directly kill child processes from a PID like so.

celery_process_pid = celery_process.pid
terminate_process(celery_process_pid )

Upvotes: 1

Related Questions