CODEWITHSUNDEEP
pythonpython-sockets
Dev Ops
Dev Ops

Reputation: 127

Sending thousands of messages through Python Socket

I have the following code:

for mydetails in allthelists:
    hostype = mydetails[0]
    graphType = mydetails[1]
    hostNodeName = mydetails[2]
    frequency = mydetails[3]
    diskName = mydetails[4]
    avgOfMultiDays = mydetails[5]
    eachEpoch = mydetails[6]

    messageA = '{hostype}.{graphType}.{hostNodeName}.{frequency}.{diskName}-altavg {avgOfMultiDays} {eachEpoch}\n'.format(**locals())
    print 'sending message:\n%s' % messageA
    sock = socket.socket()
    sock.connect((CSERVER, CPORT))
    sock.sendall(messageA)

The problem I'm having is that I have to send several thousand of "messageA" to my graphite server. The code opens and closes the socket for each message sent....which I know is terrible. Forgive me. I'm new to python.

I'm looking for a way to optimize this. I believe opening and closing a socket for each message is a very bad idea. But I cant seem to find a way around it. I would appreciate any help on this.

I was thinking along the lines of:

messageA = "blah blah blah"
messageB = "blah blah blah"
messageC = "blah blah blah"
..... and so on
#then, 
sock = socket.socket()
sock.connect((CSERVER, CPORT))
sock.sendall(messageA,messageB,messageC,.....)

But this isn't how the python socket works.

Upvotes: 0

Views: 299

Answers (1)

CryptoFool
CryptoFool

Reputation: 23129

Your initial code has a significant problem in that it doesn't close the socket after using it. If you processed thousands of messages this way, you'd be leaving open thousands of socket connections. So if you were to use your initial code as is, you'd want to put a sock.close() after the sock.sendall().

What you really want to do here depends on what the server is expecting. It isn't necessarily true for an arbitrary server listening on a socket that you can either send one message per connection or send a bunch of messages on a single connection. Since you are terminating your message with a EOL, the server could be using that to know where the end of each message is (a line is a message), in which case it could handle multiple messages. It's also possible, however, that the server expects to get only one message per connection, in which case you'd be forced to do what you're doing in your initial code block.

Assuming that you can send multiple messages on a single connection, you want to do what @GarethMa suggests...just create and connect the socket before you loop and process each message you send:

sock = socket.socket()
sock.connect((CSERVER, CPORT))
for mydetails in allthelists:
    hostype = mydetails[0]
    graphType = mydetails[1]
    hostNodeName = mydetails[2]
    frequency = mydetails[3]
    diskName = mydetails[4]
    avgOfMultiDays = mydetails[5]
    eachEpoch = mydetails[6]

    messageA = '{hostype}.{graphType}.{hostNodeName}.{frequency}.{diskName}-altavg {avgOfMultiDays} {eachEpoch}\n'.format(**locals())
    print 'sending message:\n%s' % messageA
    sock.sendall(messageA)
socket.close()

Upvotes: 2

Related Questions