CODEWITHSUNDEEP
pythonpython-2.7pickleeoferror
appy g
appy g

Reputation: 167

EOF Error on pickling in python

Here is the code that is causing this error.

sending thread:

 data = pickle.dumps (object);

receiving thread:

 self.object = pickle.loads(data) // Erroneous line

The error displayed is 

    self.object = pickle.loads(data)
EOFError

Also to add to details, this error occurs only 50% of the times. The other 50% of the times, there is no error!

Upvotes: 2

Views: 4351

Answers (1)

abarnert
abarnert

Reputation: 365717

Given the comments, I have a guess at the most likely problem, but there's at least a 50% chance I've guessed wrong, in which case… tell me and I'll delete the answer.

I'm guessing you're trying to use a stream socket as if it were a sequence of messages. This is a very common problem among network-programming novices.

Imagine the sender does something like this:

data = pickle.dumps(object);
self.sock.sendall(data)

And the receiver does something like this:

data = self.sock.recv(4096)
self.object = pickle.loads(data)

This may work 99% of the time in simple tests, but in real-world use it will not work. You will receive partial messages, or multiple messages in a single call, or some fun combination of the above (like half of message 2, all of message 3, and a third of message 4).

So, you'll pass a partial message to loads and get back an error telling you that it's not a complete pickle.

That's not because anything is broken; that's how it's supposed to work. A (TCP) socket is a stream: a sequence of bytes, not a sequence of messages. Any structure you want on top of that, you have to build into the data.

This means you have to design and implement a protocol—some way of knowing when each message is finished. The simplest protocols are probably lines (obviously only useful if a message can never have an unescaped newline) and netstrings, but anything that gives you an unambiguous way of looking at some data and saying "This is message 0, this is message 1, etc." will work.

Normally, this means appending received data onto some buffer, and looping over the messages in that buffer. For example, with lines, instead of this:

while True:
    line = sock.recv(4096)
    do_stuff(line)

… you need this:

rdbuf = ''
while True:
    rdbuf += sock.recv(4096)
    lines = rdbuf.split('\n')
    rdbuf = lines[-1]
    for line in lines[:-1]:
        dostuff(line)

If you think about it, this is no different from a file. Imagine this code:

with open('foo.data', 'wb') as f:
    f.write('123')
    f.write('45')
with open('foo.data', 'rb') as f:
    while True:
        number = f.read()

This is going to read '12345', not '123'. If you want to get the '123', you need some way of knowing to only read 3 bytes. Sticking in a length prefix, or adding whitespace as a separator, or just having external knowledge that the first number is always 3 digits long… anything works, but you have to do something.

Upvotes: 6

Related Questions