CODEWITHSUNDEEP
pythonpandasexport-to-csv
brandog
brandog

Reputation: 1527

Does Python Pandas to_csv have max number of rows?

I am attempting to export a text file into a csv. The file is very large (1.6 million rows) tab delimited file. When I export the file using to_csv it only exports 1048576 rows. Is there a maximum amount of rows that to_csv will export?

should I export the data in a different way? I would really like to be able to get it into a csv.

here is an example of my code.

import pandas as pd
import numpy as np
import os

from pandas import Series, DataFrame

pathDataEDM = "C:/Users/FILE.txt"

dataEDM = pd.read_csv(pathDataEDM, sep="\t")

dataEDM.to_csv(os.path.join(ExportDir),index=False)

Upvotes: 2

Views: 13732

Answers (2)

Jitender Reddy
Jitender Reddy

Reputation: 11

Pandas doesn't have a limit. However, most tools you use to open csv files like LibreOffice calc or excel can only display a maximum of 1048576 rows. To prove the point, try print(df) and all the 1.6 million rows would be displayed by pandas

Upvotes: 1

matusko
matusko

Reputation: 3777

I don't think there is a maximum (since it is not documented and 1.6 million is quite low for the maximum).

You can try specifying the following optional arguments (see the docs):

chunksize : int or None rows to write at a time

compression : string, optional a string representing the compression to use in the output file, allowed values are ‘gzip’, ‘bz2’, ‘xz’, only used when the first argument is a filename

Upvotes: 0

Related Questions