CODEWITHSUNDEEP
tensorflowkerasneural-networkdeep-learninggoogle-colaboratory
Noran
Noran

Reputation: 717

cannot serialize a bytes object larger than 4 GiB in deep learning

I am working on Google Colab environment to create a Siamese network using Keras to verify the images. I have used this code from GitHub. But I get an error when I tried to run the pickle.dump code: 

with open(os.path.join(save_path,"train.pickle"), "wb") as f:
    pickle.dump((X,c),f)

The error message is:

---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-7-af9d0618d385> in <module>()
      3 
      4 with open(os.path.join(save_path,"train.pickle"), "wb") as f:
----> 5         pickle.dump((X,c),f)
      6 
      7 

OverflowError: cannot serialize a bytes object larger than 4 GiB

I found some related questions in this website, but I could not find a useful answer. How can I solve this error?

Upvotes: 3

Views: 6253

Answers (1)

Bob Smith
Bob Smith

Reputation: 38579

Use pickle with protocol=4, e.g.,

pickle.dump((X,c), f, protocol=4)

Upvotes: 9

Related Questions