CODEWITHSUNDEEP
pythonpython-2.7socketstcpbuffer-overflow
PoweredByCoffee
PoweredByCoffee

Reputation: 1193

Python TCP Socket Data Sometimes Missing Parts. Socket Overflow?

Short description:

Client sends server data via TCP socket. Data varies in length and is strings broken up by the delimiter "~~~*~~~"

For the most part it works fine. For a while. After a few minutes data winds up all over the place. So I start tracking the problem and data is ending up in the wrong place because the full thing has not been passed.

Everything comes into the server script and is parsed by a different delimiter -NewData-* then placed into a Queue. This is the code:

Yes I know the buffer size is huge. No I don't send data that kind of size in one go but I was toying around with it.

class service(SocketServer.BaseRequestHandler):
    def handle(self):
        data = 'dummy'

        #print "Client connected with ", self.client_address
        while len(data):
            data = self.request.recv(163840000)
            #print data
            BigSocketParse = []
            BigSocketParse = data.split('*-New*Data-*')

            print "Putting data in queue"
            for eachmatch in BigSocketParse:
                #print eachmatch
                q.put(str(eachmatch))

            #print data
            #self.request.send(data)

        #print "Client exited"
        self.request.close()


class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
    pass

t = ThreadedTCPServer(('',500), service)
t.serve_forever()

I then have a thread running on while not q.empty(): which parses the data by the other delimiter "~~~*~~~"

So this works for a while. An example of the kind of data I'm sending:

2016-02-23 18:01:24.140000~~~*~~~Snowboarding~~~*~~~Blue Hills~~~*~~~Powder 42
~~~*~~~Board Rental~~~*~~~15.0~~~*~~~1~~~*~~~http://bigshoes.com
~~~*~~~No Wax~~~*~~~50.00~~~*~~~No Ramps~~~*~~~2016-02-23 19:45:00.000000~~~*~~~-15

But things started to break. So I took some control data and sent it in a loop. Would work for a while then results started winding up in the wrong place. And this turned up in my queue:

2016-02-23 18:01:24.140000~~~*~~~Snowboarding~~~*~~~Blue Hills~~~*~~~Powder 42
~~~*~~~Board Rental~~~*~~~15.0~~~*~~~1~~~*~~~http://bigshoes.com
~~~*~~~No Wax~~~*~~~50.00~~~*~~~No Ramps~~~*~~~2016-02-23 19:45:00.000000~~~*~

Cutting out the last "~~-15".

So the exact same data works then later doesn't. That suggests some kind of overflow to me.

The client connects like this:

class Connect(object):

    def connect(self):
        host = socket.gethostname() # Get local machine name
        #host = "127.0.0.1"
        port = 500                # Reserve a port for your service.
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        #print('connecting to host')
        sock.connect((host, port))
        return sock

    def send(self, command):
        sock = self.connect()
        #recv_data = ""
        #data = True

        #print('sending: ' + command)
        sock.sendall(command)
        sock.close()
        return

It doesn't wait for a response because I don't want it hanging around waiting for one. But it closes the socket and (as far as I understand) I don't need to flush the socket buffer or anything it should just be clearing itself when the connection closes.

Would really appreciate any help on this one. It's driving me a little spare at this point.

Updates:

I'm running this on both my local machine and a pretty beefy server and I'd be pushed to believe it's a hardware issue. The server/client both run locally and sockets are used as a way for them to communicate so I don't believe latency would be the cause.

I've been reading into the issues with TCP communication. An area where I feel I'll quickly be out of my depth but I'm starting to wonder if it's not an overflow but just some king of congestion.

If sendall on the client does not ensure everything is sent maybe some kind of timer/check on the server side to make sure nothing more is coming.

Upvotes: 1

Views: 3334

Answers (2)

tdelaney
tdelaney

Reputation: 77337

As mentioned, you are not receiving the full message even though you have a large buffer size. You need to keep receiving until you get zero bytes. You can write your own generator that takes the request object and yields the parts. The nice side is that you can start processing messages while some are still coming in

def recvblocks(request):
    buf = ''
    while 1:
        newdata = request.recv(10000)
        if not newdata:
            if buf:
                yield buf
            return
        buf += newdata
        parts = buf.split('*-New*Data-*')
        buf = parts.pop()
        for part in parts:
            yield part

But you need a fix on your client also. You need to shutdown the socket before close to really close the TCP connection

    sock.sendall(command)
    sock.shutdown(request.SHUT_RDWR)
    sock.close()

Upvotes: 2

Gerrat
Gerrat

Reputation: 29690

The basic issue is that your:

data = self.request.recv(163840000)

line is not guaranteed to receive all the data at once (regardless of how big you make the buffer).

In order to function properly, you have to handle the case where you don't get all the data at once (you need to track where you are, and append to it). See the relevant example in the Python docs on using a socket:

Now we come to the major stumbling block of sockets - send and recv operate on the network buffers. They do not necessarily handle all the bytes you hand them (or expect from them), because their major focus is handling the network buffers. In general, they return when the associated network buffers have been filled (send) or emptied (recv). They then tell you how many bytes they handled. It is your responsibility to call them again until your message has been completely dealt with.

Upvotes: 3

Related Questions