Reputation: 21
I want to load a pickle, and it worked in Python 2.7, but does not in Python 3.6 when loading it I get a memory error. I tried reformatted pickle to python 3 format, then tried to load it, but the problem persists,
The size of the pickle file is 1.1GB using Windows 10 laptop, 8GB of RAM. The pickle has 14804726 rows and 10 columns.
Any clue on how to further tackle this problem?
import pandas as pd
def readpickle(picklefile):
rawdata = pd.read_pickle(picklefile)
return rawdata
picklefile=rawdata_py3.pkl'
readpickle(picklefile)
Error:
Traceback (most recent call last):
File "<ipython-input-3-3fea1d423973>", line 1, in <module>
runfile('D:/PROJECTS/FR24/Scripts/pickletest.py', wdir='D:/PROJECTS/FR24/Scripts')
File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 11, in <module>
readpickle(picklefile)
File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 7, in readpickle
rawdata = pd.read_pickle(picklefile) #to load pickle
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 180, in read_pickle
return try_read(path, encoding='latin1')
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in try_read
lambda f: pc.load(f, encoding=encoding, compat=True))
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 149, in read_wrapper
return func(f)
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in <lambda>
lambda f: pc.load(f, encoding=encoding, compat=True))
File "c:\python36\lib\site-packages\pandas\compat\pickle_compat.py", line 212, in load
return up.load()
File "c:\python36\lib\pickle.py", line 1050, in load
dispatch[key[0]](self)
File "c:\python36\lib\pickle.py", line 1077, in load_frame
self._unframer.load_frame(frame_size)
File "c:\python36\lib\pickle.py", line 257, in load_frame
self.current_frame = io.BytesIO(self.file_read(frame_size))
MemoryError
Traceback (most recent call last):
File "<ipython-input-3-3fea1d423973>", line 1, in <module>
runfile('D:/PROJECTS/FR24/Scripts/pickletest.py', wdir='D:/PROJECTS/FR24/Scripts')
File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 11, in <module>
readpickle(picklefile)
File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 7, in readpickle
rawdata = pd.read_pickle(picklefile) #to load pickle
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 180, in read_pickle
return try_read(path, encoding='latin1')
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in try_read
lambda f: pc.load(f, encoding=encoding, compat=True))
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 149, in read_wrapper
return func(f)
File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in <lambda>
lambda f: pc.load(f, encoding=encoding, compat=True))
File "c:\python36\lib\site-packages\pandas\compat\pickle_compat.py", line 212, in load
return up.load()
File "c:\python36\lib\pickle.py", line 1050, in load
dispatch[key[0]](self)
File "c:\python36\lib\pickle.py", line 1077, in load_frame
self._unframer.load_frame(frame_size)
File "c:\python36\lib\pickle.py", line 257, in load_frame
self.current_frame = io.BytesIO(self.file_read(frame_size))
MemoryError
Upvotes: 1
Views: 1970
Reputation: 21
So in the end I figured out what the issue was, it worked on python 2.7 but not python 3.6, when comparing these I noticed that python 3.6 was 32 bit and the python 2.7 version was 64 bit. So after uninstalling and reinstalling the 64 bit python 3.6 version the pickle loaded just fine.
Upvotes: 1