CODEWITHSUNDEEP
regexscalaapache-sparkfilter
Boris Perez
Boris Perez

Reputation: 57

Regex on io.Text RDD using scala

I have a problem. I need to extract some data from a file like this:

(3269,
<page>
<title>Anarchism</title>
<ns>0</ns>
<id>12</id>
<revision>...
)
(194712,
<page>
<title>AssistiveTechnology</title>
<ns>0</ns>
<id>23</id>.. 
) etc...

This file was generated using:

val conf = new Configuration
conf.set("textinputformat.record.delimiter", "</page>")
val rdd=sc.newAPIHadoopFile("sample.bz2", classOf[TextInputFormat], classOf[LongWritable], classOf[Text], conf)
rdd.map{case (k,v) => (k.get(), new String(v.copyBytes()))}

I need to obtain the title content. Im using regex but the output file still remains empty. My code is like this:

val xx = rdd.map(x => x._2).filter(x => x.matches(".*<title>([A-Za-z]+)<\\/title>.*"))

I also try with these:

".*<title>([A-Za-z]+)</title>.*"

And using this:

val reg = ".*<title>([\\w]+)</title>.*".r
val xx = rdd.map(x => x._2).filter(x => reg.pattern.matcher(x).matches)

I create the .jar using sbt and running with spark-submit.

BTW, using spark-shell it works :S

I need your help please. Thanks.

Upvotes: 0

Views: 105

Answers (1)

Nikolay Smirnov
Nikolay Smirnov

Reputation: 36

You could use built-in Scala support for XML. Something like

import scala.xml._
rdd.map(x => (XML.loadString(x._2) \ "title").text)

Upvotes: 1

Related Questions