CODEWITHSUNDEEP
localpython-polarsazurite
Aryaman Maheshwari
Aryaman Maheshwari

Reputation: 11

Is there a way I can test polars locally using Azurite?

Azurite provides a local URI - 'http://127.0.0.1/devstoreaccount/{container_name}' to point to a file, but polars doesn't seem to have a way to interact with them. I tried using various 'storage_options' parameters but couldn't actually get it working.

I've tried -

  1. Using Localhost url provided by Azurite and plugging it in in polars.
  2. I've tried doing (1) with storage options - 'azure_storage_use_emulator'
  3. I've tried finding out the local directory where azurite creates any folders (using --hns --loose params) and seeing whether I can directly use absolute paths with polars.

Nothing worked

Upvotes: 1

Views: 78

Answers (1)

Neeraj Vernekar
Neeraj Vernekar

Reputation: 160

I tried running this code and it worked:

options = {'account_name': "devstoreaccount1", 'azure_storage_use_emulator': 'True'}

pl.read_csv("sample_1000.csv").write_delta("az://ingestion/bronze-partitioned",
                                           delta_write_options={"schema_mode": "overwrite", 'partition_by': ['City']},
                                           mode='overwrite', storage_options=options
                                           )

I am using the following versions of polars and deltalake:

polars==1.20.0
deltalake==0.24.0

However, I had to create the ingestion container by downloading Storage Explorer and logging into the emulator.

Upvotes: 0

Related Questions