CODEWITHSUNDEEP
pythonpandas
Suresh
Suresh

Reputation: 1

How to create a DataFrame of more than 700 MB data from SQL Server using python pandas

I have SQL table with more than 700 MB and keep increasing on daily basis which I have to write to a TXT file and SFTP it.

The below code is going for a time out. 

for df in pd.read_sql(query, conn, chunksize=1000000)

Is there any other way I can chunk the result and send it smoothly?

Upvotes: 0

Views: 109

Answers (1)

Umar.H
Umar.H

Reputation: 23099

your code doesn't really make any sense as you're not assigning pd.read_sql to anything

try :

chunk_size = 50000
dfs = []
for chunk in pd.read_sql("query",con=engine,chunksize=chunk_size)
    #transformation 
    dfs.append(chunk)

you can then concat the dataframe(s) and save this to a single txt file with a gzip compression since the dataframe is v.large. 

final_df = pd.concat(dfs)
final_df.to_csv('file.txt.gz',index=False,compression='gzip')

Upvotes: 1

Related Questions