CODEWITHSUNDEEP
scalaamazon-s3apache-spark
oikonomiyaki
oikonomiyaki

Reputation: 7951

Bypass last line of each file in Spark (Scala)

This question is related to this.

I am processing an S3 folder containing csv.gz files in Spark. Each csv.gz file has a header that contains column names. This has been solved by the above SO link and the solution looks like this:

val rdd = sc.textFile("s3://.../my-s3-path").mapPartitions(_.drop(1))

The problem now is that it looks like some of the files have newline ('\n') at the end (we assume we are not sure which file). So when converting the RDD to DataFrame, I'm getting some error. The question now is:

How do I get rid of the last line of each file if it is '\n'?

Upvotes: 1

Views: 1087

Answers (1)

Ramón J Romero y Vigil
Ramón J Romero y Vigil

Reputation: 17953

Why not a simple filter:

val rdd = sc.textFile("s3...").filter(line => !line.equalsIgnoreCase("\n")).mapPartition...

Or filter any empty line:

val rdd = sc.textFile("s3...").filter(line => !line.trim().isEmpty)...

Upvotes: 5

Related Questions