CODEWITHSUNDEEP
pythonpandas
Jason Melo Hall
Jason Melo Hall

Reputation: 662

Pandas Excelwriter MemoryError

import pandas

#Ignores a value is trying to be set on a copy of a slice from a DataFrame(side note if I shouldn't be doing this please let me know too)
pandas.options.mode.chained_assignment = None

#opens file
f = pandas.read_excel('.../foo.xlsx', sheetname=0)

#sort by header_number and adjust index
f = f.sort(columns=['FY15'],ascending=[0])
f.index = range(0,len(f))

#create field column
f['AB_Test'] = ''

#A/B Iteration
for i, row in enumerate(f['AB_Test']):

    if i % 2 == 0:
        f['AB_Test'][i] = 'A'
    else:
        f['AB_Test'][i] = 'B'

print f

#output to Excel
writer = pandas.ExcelWriter('output.xlsx')
f.to_excel(writer,'Sheet1')
writer.save()

f.close

Error Received

Traceback (most recent call last):
  File "C:/Users/jmhall/PycharmProjects/untitled/abtest", line 33, in <module>
    writer.save()
  File "C:\Python27\lib\site-packages\pandas\io\excel.py", line 651, in save
    return self.book.save(self.path)
  File "C:\Python27\lib\site-packages\openpyxl\workbook.py", line 265, in save
    save_workbook(self, filename)
  File "C:\Python27\lib\site-packages\openpyxl\writer\excel.py", line 186, in save_workbook
    writer = ExcelWriter(workbook)
  File "C:\Python27\lib\site-packages\openpyxl\writer\excel.py", line 71, in __init__
    self.style_writer = StyleWriter(self.workbook)
  File "C:\Python27\lib\site-packages\openpyxl\writer\styles.py", line 35, in __init__
    self._style_list = self._get_style_list(workbook)
  File "C:\Python27\lib\site-packages\openpyxl\writer\styles.py", line 43, in _get_style_list
    uniqueStyles = dict((id(style), style) for style in worksheet._styles.values()).values()
MemoryError

Process finished with exit code 1

I ran the code below from a different solution I found and got the results below.

Code

pandas.__version__
pandas.show_versions(as_json=False)

Results

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 32
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.16.2
nose: None
Cython: None
numpy: 1.9.2
scipy: None
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.4
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: 1.8.6
xlrd: 0.9.4
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None

Upvotes: 2

Views: 6203

Answers (2)

Mike
Mike

Reputation: 11

I think the reason is that you have a 32bit version installed. This allows accessing only a few Gigs of Memory. Try the 64bit version of python.

Upvotes: 1

Andrea Dusza
Andrea Dusza

Reputation: 2110

Can it be that your input file is too big? (Judging by the MemoryError.) In this case, you could try breaking the data up to smaller chunks instead of trying to write it all at once to an .xlsx file.

Upvotes: 5

Related Questions