CODEWITHSUNDEEP
javaapache-sparkrdd
volity
volity

Reputation: 93

How to join two special RDDs?

One is 

rdd1 : JavaPairRDD<Tuple2<String,String>,Integer>

another is

rdd2 : JavaPairRDD<String,Integer>

I want to join rdd1 and rdd2 where Tuple2._1 in rdd1 equals to the key in rdd2. For example, (("a","b"),1) and ("a",2) will generate (("a","b"),1,2). When I map rdd1 to:

rdd3 : JavaPairRDD<String, Tuple2<String, Integer>>

and try to use rdd3.join(rdd2) , it arose “can only concatenate tuple (not "str") to tuple”. Is there a solution to join rdd1 and rdd2 and get the results I want?

Upvotes: 0

Views: 172

Answers (1)

user6022341
user6022341

Reputation:

Map rdd1 to:

JavaPairRDD<String, Tuple2<Tuple2<String,String>,Integer>>

with something like:

x -> new Tuple2(x._1._1, x)

use standard join and map once again to have desired result

Upvotes: 1

Related Questions