CODEWITHSUNDEEP
pythonpython-3.xpyspark
Swati B
Swati B

Reputation: 63

Fetch all the file names from blob storage which came on particular day using python/pyspark

I have multiple files in my blob storage , I want to fetch only the file names in form of a list which came on certain day.

for ex: blob strage has all files like

files  Modified
abc    8/22/2022, 1:34:16 AM
bdg    9/12/2022, 2:34:12 AM
hgf    9/12/2022, 5:34:12 AM

i need to fetch all the files which came on 9/12/2022 how to achieve this using python/pyspark?

Upvotes: 3

Views: 753

Answers (1)

Assaf Segev
Assaf Segev

Reputation: 391

I'm not sure what's file name format, but try this -

import pyspark.sql.functions as F 

spark.read.format(file_format).load(path).withColumn('file_name', F.input_file_name())

Then filter the df by the relevant date.

Upvotes: 2

Related Questions