CODEWITHSUNDEEP
pythonjsonpicklestatsmodels
Jesse Blocher
Jesse Blocher

Reputation: 523

Saving list of objects in Python

I've run a regression 5,693 times and want to save the output, since it took several hours to run. I've captured it in a list called res, and the object (if it matters) is a MarkovRegressionResultsWrapper object from package statsmodels.

I thought the way to go was pickle. I'm saving to a private directory for my own use, so security isn't an issue, and JSON doesn't seem to work for objects (I'm new, so perhaps this is wrong?).

Here is an example I found that works fine:

import pickle
a = ['test value','test value 2','test value 3']

file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb') 

# this writes the object a to the
# file named 'testfile'
pickle.dump(a,fileObject)   

# here we close the fileObject
fileObject.close()

However, when I use the exact same code, but save my list res, it gives an error:

file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb') 

# this writes the object a to the
# file named 'testfile'
pickle.dump(res,fileObject)   

# here we close the fileObject
fileObject.close()

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-43-ab4800ac1a51> in <module>()
      7 # this writes the object a to the
      8 # file named 'testfile'
----> 9 pickle.dump(res,fileObject)
     10 
     11 # here we close the fileObject

OSError: [Errno 22] Invalid argument

I'm using Python 3.6 with Jupyter Notebook on a Macbook Pro. Both a and res are of type list, so the only thing that is different is what the list contains. Why am I getting this error? Is this the best way to save this list of objects or should I be doing something different?

Upvotes: 0

Views: 2054

Answers (1)

Jesse Blocher
Jesse Blocher

Reputation: 523

@ChristianDean provided the answer in the comments. This is related to a known bug in pickle in Python 3.6 on Mac OSX only. Python 3 - Can pickle handle byte objects larger than 4GB?

Upvotes: 2

Related Questions