CODEWITHSUNDEEP
apache-sparkpysparkcompressionvirtualenvlzo
Abdullah Odibat
Abdullah Odibat

Reputation: 21

How to read LZO compressed files in Pyspark

I am using PySpark 3.1.2 with virtual environment.

I am trying to read files compressed with lzo but i cant find proper documentation on how to do that, i understand that for licensing issue, the lzo codec needs to be added manually to spark. But i dont find a step by step documentation to do that.

I already checked the question: Read Lzo file in PySpark but id did not really help me, as i would like to add the codec in spark so im able to read directory of lzo files same as i do with parquet and json.

Any help will be appreciated :)

Upvotes: 1

Views: 568

Answers (0)

Related Questions