CODEWITHSUNDEEP
pythonpandasparquetbigdata
ZeusofCode
ZeusofCode

Reputation: 83

Converting .parquet file to CSV using Pyarrow

I have a .parquet file and I am using PyArrow. I converted the .parquet file into a table using the following code:

import pyarrow.parquet as pq
import pandas as pd
filepath = "xxx"  # This contains the exact location of the file on the server
from pandas import Series, DataFrame
table = pq.read_table(filepath)

Performing table.shape returned (39014 rows, 19 columns).

The schema of the table is:

col1: int64 not null
col2: string not null
col3: string not null
col4: int64 not null
col5: string not null
col6: string not null
col7: int64 not null
col8: int64 not null
col9: string not null
col10: string not null
col11: string not null
col12: string not null
col13: string not null
col14: string not null
col15: string not null
col16: int64 not null
col17: int64 not null
col18: int64 not null
col19: string not null

When performing p = table.to_pandas() I get the following error:

ImportError: cannot import name RangeIndex

How do I convert this parquet file into a dataframe and then CSV ? Please help. Thank you.

Upvotes: 4

Views: 8352

Answers (1)

SSingh
SSingh

Reputation: 209

Try following:

import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
import pyodbc

def read_pyarrow(path, nthreads=1):
    return pq.read_table(path, nthreads=nthreads).to_pandas()

path = './test.parquet'
df1 = read_pyarrow(path)

df1.to_csv(
    './test.csv',
    sep='|',
    index=False,
    mode='w',
    line_terminator='\n',
    encoding='utf-8')

Upvotes: 3

Related Questions