CODEWITHSUNDEEP
apache-sparkapache-spark-sqlspark-streamingdatabricks
RichaDwivedi
RichaDwivedi

Reputation: 343

Caused by: IllegalArgumentException: Wrong basePath for the root path

I am writing a dataframe which has column id with .partitionBy("id") on Id column

s3a://bucket/abc/id=0 
s3a://bucket/abc/id=1

but while reading from s3a://bucket/abc/id=1 I am getting below error -

Caused by: IllegalArgumentException: Wrong basePath s3a://bucket/abc/id=1 for the root path: s3a://bucket/abc/id=0/part-1.c000.snappy.parquet

can anyone please help, I am stuck with this

Upvotes: 0

Views: 449

Answers (1)

Nathan Chong
Nathan Chong

Reputation: 11

This is a shot in the dark, but are you using streaming dataframes? You might be using the same checkpoint for multiple different source streams accidentally!

Upvotes: 1

Related Questions