CODEWITHSUNDEEP
apache-sparkpysparkspark-streaming
Khan Hafizur Rahman
Khan Hafizur Rahman

Reputation: 105

Found nothing in _spark_metadata

I am trying to read CSV files from a specific folder and write same contents to other CSV file in a different location on the local pc for learning purpose. I can read the file and show the contents on the console. However, if I want to write it to another CSV file at the specified output directory I get a folder named "_spark_metadata" which contain nothing inside.

I paste the whole code here step by step:

creating Spark Session:

spark = SparkSession \
.builder \
.appName('csv01') \
.master('local[*]') \
.getOrCreate();

spark.conf.set("spark.sql.streaming.checkpointLocation", <String path to checkpoint location directory> )
userSchema = StructType().add("name", "string").add("age", "integer")

Read from CSV file

df = spark \
.readStream \
.schema(userSchema) \
.option("sep",",") \
.csv(<String path to local input directory containing CSV file>)

Write to CSV file

df.writeStream \
.format("csv") \
.option("path", <String path to local output directory containing CSV file>) \
.start()

In "String path to local output directory containing CSV file" I only get a folder _spark_metadata which contains no CSV file.

Any help on this is highly appreciated

Upvotes: 0

Views: 962

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191681

You don't use readStream to read from static data. You use that to read from a directory where files are added into that folder.

You only need spark.read.csv

Upvotes: 1

Related Questions