CODEWITHSUNDEEP
pythonpandas
littlely
littlely

Reputation: 1428

Converting generator from read_sql in pandas to dataframe has failed

I want to read data from my oracle, I use the pandas's read_sql and set the parameter chunksize=20000, 

from sqlalchemy import create_engine
import pandas as pd
engine = create_engine("my oracle")
df = pd.read_sql("select clause",engine,chunksize=20000)

It returns a iterator, and I want to convert this generator to a dataframe usingdf = pd.DataFrame(df), but it's wrong, How can the iterator be converted to a dataframe?

Upvotes: 0

Views: 2233

Answers (1)

giser_yugang
giser_yugang

Reputation: 6176

This iterator can be concatenated, then it return a dataframe:

df = pd.concat(df)

You can view pandas.concat document.

If you can't use concat directly, try the following:

gens = pd.read_sql("select clause",engine,chunksize=20000)
dflist = []
for gen in gens:
    dflist.append(gen)
df = pd.concat(dflist)

Upvotes: 2

Related Questions