CODEWITHSUNDEEP
pythonmultithreadingerror-handlingautomation
Drew
Drew

Reputation: 229

Is there a good way to split a python program into independent modules?

I'm trying to do some machinery automation with python, but I've run into a problem.

I have code that does the actual control, code that logs, code the provides a GUI, and some other modules all being called from a single script.

The issue is that an error in one module halts all the others. So, for instance a bug in the GUI will kill the control systems.

I want to be able to have the modules run independently, so one can crash, be restarted, be patched, etc without halting the others.

The only way I can find to make that work is to store the variables in an SQL database, or files or something.

Is there a way for one python script to sort of ..debug another? so that one script can read or change the variables in the other? I can't find a way to do that that also allows to scripts to be started and stopped independently.

Does anyone have any ideas or advice?

Upvotes: 1

Views: 232

Answers (2)

bzdjamboo
bzdjamboo

Reputation: 565

You want multiply independent processes, and you want them to talk to each other. Hence: read what methods of inter-process communication are available on your OS. I recommend sockets (generic, will work over a n/w and with diff OSs). You can easily invent a simple (maybe http-like) protocol on top of TCP, maybe with json for messages. There is a bunch of classes coming with Python distribution to make it easy (SocketServer.ThreadingMixIn, SocketServer.TCPServer, etc.).

Upvotes: 0

Tim McNamara
Tim McNamara

Reputation: 18385

A fairly effective way to do this is to use message passing. Each of your modules are independent, but they can send and receive messages to each other. A very good reference on the many ways to achieve this in Python is the Python wiki page for parallel processing.

A generic strategy

Split your program into pieces where there are servers and clients. You could then use middleware such as 0MQ, Apache ActiveMQ or RabbitMQ to send data between different parts of the system.

In this case, your GUI could send a message to the log parser server telling it to begin work. Once it's done, the log parser will send a broadcast message to anyone interested telling the world the a reference to the results. The GUI could be a subscriber to the channel that the log parser subscribes to. Once it receives the message, it will open up the results file and display whatever the user is interested in.

Serialization and deserialization speed is important also. You want to minimise the overhead for communicating. Google Protocol Buffers and Apache Thrift are effective tools here.

You will also need some form of supervision strategy to prevent a failure in one of the servers from blocking everything. supervisord will restart things for you and is quite easy to configure. Again, it is only one of many options in this space.

Overkill much?

It sounds like you have created a simple utility. The multiprocessing module is an excellent way to have different bits of the program running fairly independently. You still apply the same strategy (message passing, no shared shared state, supervision), but with different tactics.

Upvotes: 1

Related Questions