Sahand Pourjavad
Reputation: 29
How to load all csv files in a folder with pyspark
I have a folder which has
Sales_December.csv
Sales_January.csv
Sales_February.csv
etc.
How can i make pyspark read all of them into 1 dataframe?
Upvotes: 0
Views: 211
Answers (1)
Gprj
Reputation: 32
- create an empty list
- read your csv files one by one and append DataFrames to the list
- use
reduce(DataFrame.unionAll, <list>)to combine them into one single DataFrame
Upvotes: 1
Related Questions
- pyspark read multiple csv files at once
- Load CSV file with PySpark
- How to properly import CSV files with PySpark
- How to read multiple CSV (leaving out specific ones) from a nested directory in PySpark?
- ¿How do I read multiple files from multiple folders in Python
- Pyspark: how to read multiple csv from different folders?
- how to read csv file in pyspark?
- PYSPARK - How to read all csv files in all subfolders in S3?
- Spark: reading many files with read.csv
- How to load only first n files in pyspark spark.read.csv from a single directory