CODEWITHSUNDEEP
pysparkcompressionazure-synapseazure-data-lake-gen2tar.gz
blessy154
blessy154

Reputation: 11

decompress tarfile from adls gen2 to synapse notebook

I have a tar.gz file in adls gen2 location and i need to load and decompress that file in the synapse notebook using pyspark

for file in mssparkutils.fs.ls("synfs:/13/Input/ZippedFolder"):
   if file.name.endswith('.gz'):
       #print(file)
       zipped = gzip.open(io.StringIO(file.name),mode='r',compresslevel=9,encoding=None,errors=None,newline=None)
       print(zipped)
      
       #with gzip.GzipFile(builtins.open(zipped, 'rb')) as f:
       with gzip.GzipFile(zipped, 'rb',compresslevel=9, fileobj=None, mtime=None) as f:
        gzip.decompress(f)

this results the following error

expected str, bytes or os.PathLike object, not GzipFile

I think i need to pass rdd objects and need assistance

Upvotes: 1

Views: 330

Answers (0)

Related Questions