CODEWITHSUNDEEP
scalaapache-sparkfilterparametersrdd
Lauren
Lauren

Reputation: 337

"Item does not take parameters" when filtering RDD - scala, Apache Spark

I am working with a csv file in Apache Spark. I created the RDD and I want to obtain separate RDDs for each month, so I am filtering by the date, which is expressed as 201601, 201602, etc... 

case class Item(date: String, id: String, classification: String, description: String, algoIndex: String, stratumIndex: String, itemIndex: String, allGmIndex: String, gmRaIndex: String, coicopWeight: String, itemWeight: String, cpihCoicopWeight: String)

val quarter1 = sc.textFile ("examples/Q1.csv")

val q1 = quarter1 map {i => {
        val x = i.split(",")
        Item(x(0), x(1), x(2), x(3), x(4), x(5), x(6), x(7), x(8), x(9), x(10), x(11))
        }
    }

val Jan = q1.filter {x=> x(0) == "201601"}
val Feb = q1.filter {x=> x(0) == "201602"}
val Mar = q1.filter {x=> x(0) == "201603"}

The last three lines cause the error "Item does not take parameters", in relation to the following bit:

{x=> x(0) == ..}

How can I fix this? What did I do wrong in my class? Thank you a lot in advance! :)

Upvotes: 2

Views: 458

Answers (2)

Andrey Tyukin
Andrey Tyukin

Reputation: 44957

Your xs in the function literals are of type Item. Item does not have an apply(i: Int) method. Instead, Item has a bunch of named member variables.

Try something like this instead:

val Jan = q1.filter {x => x.date == "201601"}

It can be abbreviated to:

val Jan = q1.filter (_.date == "201601")

Upvotes: 2

user9289446
user9289446

Reputation: 41

You use a case class, so you cannot use apply syntax. Use the names you defined:

q1.filter { x => x.date == "201601" }

or 

q1.filter { _.date == "201601" }

Upvotes: 4

Related Questions