CODEWITHSUNDEEP
pythontensorflowserializationpickledill
eqzx
eqzx

Reputation: 5579

Why does pickling a tensorflow Tensor fail?

Here's a snippet that will succeed in serializing with dill, but fail with pickle. It is surprising that Tensor objects aren't natively pickleable. Is this a fundamental limitation of thread-aware Tensors, or is it just not implemented?

import dill
import pickle
import tensorflow as tf

dill.dumps(tf.zeros((1,1)))
print("Dill succeeded")
pickle.dumps(tf.zeros((1,1)))
print("Pickle succeeded")

Output:

$ python foo.py
Dill succeeded
Traceback (most recent call last):
  File "foo.py", line 7, in <module>
    pickle.dumps(tf.zeros((1,1)))
TypeError: can't pickle _thread.lock objects

Upvotes: 4

Views: 6684

Answers (2)

Mike McKerns
Mike McKerns

Reputation: 35247

The reason why dill can serialize these objects, but not pickle? Simple answer is that pickle cannot serialize most objects in python, the thread.lock object included. If you want to serialize one of these objects, use an advanced serialization library like dill. As to exactly why pickle can't, I think originally it stems from the implementation of the GIL and the frame object rendering some objects unserializable, and thus there was no drive to serialize everything in the language. There's always been talk about security issues stemming from serialization of all python objects, but I think that's a red herring. Not having full language serialization limits the ability to operate in parallel computing, so hopefully pickle will learn from dill how to serialize more objects.

Upvotes: 3

evantkchong
evantkchong

Reputation: 2606

I find that I am unable to reproduce the output of eqzx's script and serializing fails for both dill and pickle.

While this question is several years old, I believe the issue wasn't with pickle but rather the fact that tensors in tensorflow 1.x are not evaluated immediately. They are instead evaluated when their graph is executed. (In tensorflow 2.0 eager execution is enabled by default so you don't really have to deal with this paradigm).

Using tf.enable_eager_execution() or evaluating a tensor within a tf.Session() context

The following script doesn't throw an error when using either Pickle or Dill:

import tensorflow as tf
tf.enable_eager_execution()
import pickle 
# import dill

n = tf.zeros((1,1))
pickle.dumps(n)
# dill.dumps(n)

print('Success')

Same script using Session.run() instead doesn't throw an error:

import tensorflow as tf
import pickle
# import dill

with tf.Session() as sess:
    n = sess.run(tf.zeros((1,1)))
pickle.dumps(n)
# dill.dumps(n)

print('Success')

Upvotes: 2

Related Questions