CODEWITHSUNDEEP
pythonpandasdataframe
stockersky
stockersky

Reputation: 1571

Pandas DataFrame.to_csv last column missing when field is empty

I am extracting data from a database into a csv file.

I realize that each time the last field of a row in Null, then DataFrame.to_csv omit it. This only happens when the empty field is at the last position.

Here is an exemple :

dframe_iterator = pandas.read_sql_query(request, engine, chunksize=1000)
for i, dataframe in enumerate(dframe_iterator):   
        dataframe.to_csv('file.csv', index=False, header=True, sep='|', mode='a', encoding='utf-8', date_format='%d/%m/%Y')

Let say one n-uplet returned by the sql query contains 2 Null values :

'blabla','blabla',Null, 'blabla', Null

Then, in the csv file, i get :

blabla|blabla||blabla

You can see that the first Null field is there (||) but the second Null filed is omitted.

I would expect this :

blabla|blabla||blabla|

Do you have any idea how to perform this? Another application is expecting as much fields as returned by the sql query.

Thanks!

Upvotes: 1

Views: 1640

Answers (2)

stockersky
stockersky

Reputation: 1571

Hemm, well, I apologie but my question was wrong.

Actually, the pandas behaviour is perfectly good :

'blabla','blabla',Null, 'blabla', Null

would be :

blabla|blabla||blabla|

I have been troubled by a dataset having lots of Null field at last positions. Working on a different dataset made me realize this. And also wrong client specs which would expect blabla|blabla||blabla||

I really do apologie for being stupid and too fast to post.

Upvotes: 2

Eugene Lisitsky
Eugene Lisitsky

Reputation: 12855

Have you tried parameter na_rep? Doc

  na_rep : string, default ‘’
       Missing data representation

Upvotes: 0

Related Questions