CODEWITHSUNDEEP
pythonwindowspython-3.x
Robbie
Robbie

Reputation: 304

method of sending data from multiple computers too a central location in real-time using python

I'm looking for a reliable method of sending data from multiple computers to one central computer that will receive all the data, process and analyse it. All computers will be on the same network.

I will be sending text from the machines so ideally it will be a file that I need sending, maybe even an XML file so I can parse it into a database easily.

The main issue I have is I need to do this in near enough real-time. For example if an event happens on pc1, I need to be able to send that event plus any relevant information back to a central pc so it can be used and viewed almost immediately.

The plan is to write a python program that possibly acts as a sort of client that detects the event and sends it off to a server program on the central pc.

Does anyone have suggestions on a reliable way to send data from multiple computers to one central computer/server all on the same network and preferably without going out onto the web.

Upvotes: 1

Views: 855

Answers (4)

user1277476
user1277476

Reputation: 2909

A database is an interesting answer.

Personally, I'd probably use TCP with a new session for each message.

Writing TCP client/server code to be reliable is greatly facilitated by something like http://stromberg.dnsalias.org/~strombrg/bufsock.html - otherwise you tend to get seemingly-inscrutable problems under high load and/or long network paths.

Once you've got TCP going, you could create a new file for each message in a filesystem. I find this simpler than using a database, but whatever. If you put the time, hostname and a nonce in the filenames, you should have no difficulty with concurrency.

A 3rd approach would be a REST interface. I don't have much experience with this, but it's very popular these days.

Upvotes: 0

panofish
panofish

Reputation: 7889

There are many possible solutions, but my recommendation is to solve this with a database. I prefer MySQL since it is free and easy to setup. It is immediate and you can avoid simultaneous update file locking problems because Innodb feature of MySQL automatically handles row locking. It's actually easier to setup a database than to try to write your own solution using files or other communication mechanisms (unless you already have experience with other techniques). Multiple computers is also not an issue and security is also built-in.

Just setup your MySQL server and write a client application to update the data from multiple computer to your server. Then you can write your server application to process the input and that program can reside on your MySQL server or any other computer.

Python is an excellent programming language and provides full support through readily available modules for MySQL. This solution is also scalable, because you can start with basic command line programs... then create desktop user interface with pyqt and later you could add web interfaces if you desire.

Upvotes: 2

Serge Ballesta
Serge Ballesta

Reputation: 149115

I assume you are working only on a secure lan, since you do not speak about security, but only low latency. In fact there are many solutions each with its advantages and drawbacks.

  1. Simple messages using UDP. Very low overhead on network, client and server. Drawback : in UDP you can never be sure that a message cannot be lost. Use case : small pieces of information that can be generated at any time with high frequency, and it is not important if one is lost
  2. Messages over a pre-established TCP connection. High overhead on server, client and network, because each client will establish and maintain a connection to the server, and the server will simultaneouly listen to all its clients. Drawback: need to reopen a connection if it breaks, server will have to multiplex io, you have to implement a protocol to separate messages. Use case : you cannot afford to loose any message, they must be sent as soon as possible and each client must serialize its own messages.
  3. Messages over TCP, each message will use its own connection. Medium overhead. Drawback : as a new connection is established per message, it may introduce a latency and overhead in high frequency events. Use case : low frequency events, but a single client PC may send simultaneously multiple messages

Upvotes: 0

Mark Setchell
Mark Setchell

Reputation: 207708

Netcat over TCP for reliability, low overhead and simplicity.

Upvotes: 0

Related Questions