CODEWITHSUNDEEP
pythonmultithreadinglocking
eran
eran

Reputation: 15136

Communicating between one Read and one Write process in python

I am looking for a way to pass information safely (!) between two processes of python scripts.

One process reads only, and the other writes only. The data I need to pass is a dictionary.

The best solution I found so far is a local SQL server (since the data itself is kind of a table) and I assume SQLite will handle the transaction safely.

Is there a better way, maybe a module to lock a file from being read while it is written to and vice versa?

I am using linux ubuntu 11.10, but a cross platform solution will be welcome.

Upvotes: 0

Views: 161

Answers (2)

Roland Smith
Roland Smith

Reputation: 43495

For one-way communication you could use e.g. multiprocessing.Queue or multiprocessing.queues.SimpleQueue.

Shared memory is also an option using multiprocessing.Array. But you'd have to split up the dictionary in at least two arrays (keys and values). This will only work well if all the values are of the same basic type.

The good point about both multiprocessing.Queue and multiprocessing.Array is that they are both protected by locks internally, so you don't have to worry about that.

Upvotes: 1

Aaron Digulla
Aaron Digulla

Reputation: 328584

To transfer data between processes, you can use pipes and sockets. Python uses pickling to convert between objects and byte streams.

To transfer the data safely, you need to make sure that the data is transferred exactly once. Which means: The destination process needs to be able to say "I didn't get everything, send it again" while the sender needs some form of receipt.

To achieve this, you should add a "header" to the data which gives it a unique key (maybe a timestamp or a hash). Both receiver and sender can then keep a list of those they have sent/seen to avoid processing data twice.

Upvotes: 3

Related Questions