CODEWITHSUNDEEP
pythonazure-data-lake
Juanita Smith
Juanita Smith

Reputation: 169

Write Python Dataframe to CSV file directly in Azure Datalake

I have imported an excel file into a pandas dataframe and have completed the data exploration and cleaning process.

I now want to write the cleaned dataframe to csv file back to Azure DataLake, without saving it first as a local file. I am using pandas 3.

My code looks like this:

token = lib.auth(tenant_id = '', 
                 client_secret ='', 
                 client_id = '')

adl = core.AzureDLFileSystem(token, store_name)

with adl.open(path='Raw/Gold/Myfile.csv', mode='wb') as f:
    **in_xls.to_csv(f, encoding='utf-8')**
    f.close()

I get the following dump in statement in bold.

TypeError: a bytes-like object is required, not 'str'

I also tried but without any luck

with adl.open(path='Raw/Gold/Myfile.csv', mode='wb') as f:
    with io.BytesIO(in_xls) as byte_buf:
        byte_buf.to_csv(f, encoding='utf-8')
        f.close()

I am getting the below error:

TypeError: a bytes-like object is required, not 'DataFrame'

Any ideas/tips will be much appreciated

Upvotes: 8

Views: 12671

Answers (2)

Sebastian Bujak
Sebastian Bujak

Reputation: 261

For anyone visiting this question after 2020: this issue was solved with pandas 1.2.0 (support-for-binary-file-handles-in-to-csv)

With pandas 1.2.0 onward this works:

adl_fs = AzureDLFileSystem(your_adl_creds, store_name='your_store_name')
with adl_fs.open('/path/file.csv', mode='wb') as adl_file:
   some_pandas_df.to_csv(adl_file)

The proposed solution of creating intermediate string variable was valid back in 2017 when it was written, but it comes with a caveat of unnecessary consumption of memory, which can be a problem when working with big files. (This is how I came across this question BTW).

Upvotes: 0

ShowMeTheData
ShowMeTheData

Reputation: 111

I got this working with pandas the other day with python 3.X. This code runs on an on premise machine and connects to the azure data store in the cloud.

Assuming df is a pandas dataframe you can use the following code:

adl = core.AzureDLFileSystem(token, store_name='YOUR_ADLS_STORE_NAME')
      #toke is your login token that was created by whatever ADLS login method you decided.
      #Personally I use the ServiceProvider login
df_str = df.to_csv()
with adl.open('/path/to/file/on/adls/newfile.csv', 'wb') as f:
    f.write(str.encode(df_str))
    f.close()

This key is converting the dataframe to a string and than using the str.encode() function.

Hope this helps.

Upvotes: 10

Related Questions