CODEWITHSUNDEEP
pythondjango
FiR3WaLL
FiR3WaLL

Reputation: 33

Python parallel processing from client-server application

I have a web application (django) that stores in mysql database PID numbers of processes from remote Linux machine. I designed a simple server-client application that talking to remote server and getting me some data about given PID number (cpu%, mem%) ... this data is from 5s interval.

But there is a performance problem .... I have 200 pids to check and every of them takes ~5 sec and they are processing in the for loop. So I have situation where I`m waiting 200*5 sec minimum

Can somebody advise me how to make it parallel processing? So my application will be able to fetch for example 50 pids at one time ... I believe python client - server library can handle multiple requests coming to the server.

I want to archive something like:

for pid in my_200_pid_list:
  // Some parallel magic to not wait and pass another 49...
  result[pid] = askforprocess(pid)

My client code:

def askforprocess(processpid):
    #Create TCP/IP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect on host and port provided in command line arguments
    server_address = ('172.16.1.105', int('5055'))
    sock.connect(server_address)

    # Send the data
    try:
            message = processpid
            sock.sendall(message)

            data = sock.recv(2048)

    finally:
            sock.close()


    return data

Upvotes: 0

Views: 1158

Answers (2)

notorious.no
notorious.no

Reputation: 5107

In general, it's best to do stuff like this using a single thread when possible. You just have to make sure your functions don't block other functions. The builtin lib that comes to mind is select. Unfortunately, it's a bit difficult to explain and I haven't used it in quite some time. Hopefully this link will help you understand it http://pymotw.com/2/select/.

You can also use the multiprocessing lib and poll each pid in a separate thread. This can be very difficult to manage if you plan to scale out further! Use threads only as a last resort (this is my usual rule of thumb when it comes to threads). https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing

from multiprocessing import Process

def askforprocess(processpid):
    #Create TCP/IP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect on host and port provided in command line arguments
    server_address = ('172.16.1.105', int('5055'))
    sock.connect(server_address)

    # Send the data
    try:
            message = processpid
            sock.sendall(message)
            data = sock.recv(2048)
    finally:
            sock.close()
    return data

if __name__ == '__main__':
    info('main line')
    p = Process(target=askforprocess, args=(processpid,))
    p.start()

Lastly, there's Twisted library which is probably the most difficult to understand, but defiantly makes concurrent (not necessarily parallel) functions easy to write. Only bad thing is you'd probably have to rewrite your entire app in order to use Twisted. Don't be put off by this fact, try to use it if you can.

Hope that helps.

Upvotes: 1

Mykhaylo Kopytonenko
Mykhaylo Kopytonenko

Reputation: 953

Use threads to process your requests in parallel: https://docs.python.org/2/library/threading.html

Upvotes: 0

Related Questions