CODEWITHSUNDEEP
pythontwisted
Nathan
Nathan

Reputation: 2990

Best way for client to fire off separate process without blocking client/server communication

The end result I am trying to achieve is allow a server to assign specific tasks to a client when it makes it's connection. A simplified version would be like this

  1. Client connects to Server
  2. Server tells Client to run some network task
  3. Client receives task and fires up another process to complete task
  4. Client tells Server it has started
  5. Server tells Client it has another task to do (and so on...)

A couple of notes

At first, I was going to try threading, but I have heard python doesn't do threading correctly (is that right/wrong?)

Then it was thought to fire of a system call from python and record the PID. Then send certain signals to it for status, stop, (SIGUSR1, SIGUSR2, SIGINT). But not sure if that will work, because I don't know if I can capture data from another process. If you can, I don't have a clue how that would be accomplished. (stdout or a socket file?)

What would you guys suggest as far as the best way to handle this?

Upvotes: 2

Views: 151

Answers (2)

Glyph
Glyph

Reputation: 31910

Use spawnProcess to spawn a subprocess. If you're using Twisted already, then this should integrate pretty seamlessly into your existing protocol logic.

Upvotes: 4

Andrew Gorcester
Andrew Gorcester

Reputation: 19983

Use Celery, a Python distributed task queue. It probably does everything you want or can be made to do everything you want, and it will also handle a ton of edge cases you might not have considered yet (what happens to existing jobs if the server crashes, etc.)

You can communicate with Celery from your other software using a messaging queue like RabbitMQ; see the Celery tutorials for details on this.

It will probably be most convenient to use a database such as MySQL or PostgreSQL to store information about tasks and their results, but you may be able to engineer a solution that doesn't use a database if you prefer.

Upvotes: 0

Related Questions