CODEWITHSUNDEEP
pythonpandaspyarrow
Luis
Luis

Reputation: 318

Pyarrow is slower than pandas for csv read in

I am trying to open some csv files with python 3.9. I was puzzled by timing pandas vs pyarrow :

enter image description here

Is it normal that pyarrow takes that much time? I thought that it was commonly agreed that it should be much faster. These csv files have not been created using pandas and their size is about 5GB (they come from the Twitter API). What could be the reasons that lead to this result?

Upvotes: 1

Views: 1336

Answers (1)

karim
karim

Reputation: 121

Pyarrow and Macbook Silicon

If you are using a Mac computer with Apple Silicon, please make sure that you are using the arm64 version of python and not x86_64.

Upvotes: 1

Related Questions