CODEWITHSUNDEEP
pythonmultiprocessingterminate
Matt Seymour
Matt Seymour

Reputation: 9395

Mulitprocessing, terminate and corrupt queues

Whilst looking at the python doc for multiprocessing.terminate() I came across the following:

Terminate() If this method is used when the associated process is using a pipe or queue then the pipe or queue is liable to become corrupted and may become unusable by other process. Similarly, if the process has acquired a lock or semaphore etc. then terminating it is liable to cause other processes to deadlock.

Which basically says if you terminate a process which is using a queue, pipe or similar you run the risk of the structure becoming corrupt.

I have a couple of questions about this,

  1. What will happen to another process trying to retrieve the data from the PIPE, Queue or similar if a corruption occurs?
  2. How can a process check to see if there is corruption?
  3. Can the deadlock be resolved in any way if you know another process has been terminated?

I understand you should always try to not use terminate, but this is for that situation where you cannot do anything else but this

Upvotes: 6

Views: 729

Answers (2)

Matt Seymour
Matt Seymour

Reputation: 9395

Okay, so it was not the nicest solution but I have firstly handled the exceptions as best as possible. I did not want to run the risk of having corruption so I now restart the application in an instance where there is the possibility of a corruption occurring to reduce the chance of possible issues.

Thanks @MarwanAlsabbagh for your suggestion.

Upvotes: 1

Marwan Alsabbagh
Marwan Alsabbagh

Reputation: 26788

You could add checksums to the blocks of data you pass around and check them to confirm no data corruption occurred. This is a common technique in any data communication that has a risk of data corruption. You could look at hashlib for this and use something like md5 or crc32 checksums.

Upvotes: 3

Related Questions