CODEWITHSUNDEEP
scalaapache-spark
Khumar
Khumar

Reputation: 336

filter and store the result using spark

I am having file which contain data like below:

100|hyd|xxx|32
101|chn|yyy|98
103|chn|abc|87
104|hyd|nbx|56

Here I want to filter the data based on location(hyd,chn) and store it in a text file. I tried the below code.

val file=sc.textFile("/home/cloudera/abc.txt")
val file2=file.map(line=>line.split("\\|"))
val file3 = file2.filter(line=>line.apply(1).matches("hyd")).saveAsTextFile("/home/cloudera/hyd")

When I check the /home/cloudera/hyd/part-00000 path data is stored in object format. [Ljava.lang.String;@679e1175

I want the data to be stored in plain text format.

100|hyd|xxx|32
104|hyd|nbx|56

Thank you.

Upvotes: 0

Views: 82

Answers (1)

Shivansh
Shivansh

Reputation: 3544

You are just missing one thing converting the list to String!

This can be easily done in this way:

val file=sc.textFile("/home/cloudera/abc.txt")
val file2=file.map(line=>line.split("\\|"))
val file3 = file2.filter(line=>line.apply(1).matches("hyd")).map(line=>line.mkString("|")).saveAsTextFile("/home/cloudera/hyd")

Upvotes: 2

Related Questions