CODEWITHSUNDEEP
scalahadoopmapreducecascadingscalding
samthebest
samthebest

Reputation: 31513

Write to multiple outputs by key Scalding Hadoop, one MapReduce Job

How can you write to multiple outputs dependent on the key using Scalding(/cascading) in a single Map Reduce Job. I could of course use .filter for all the possible keys, but that is a horrible hack, which will fire up many jobs.

Upvotes: 6

Views: 3739

Answers (3)

morazow
morazow

Reputation: 261

There is TemplatedTsv in Scalding (from version 0.9.0rc16 and up), exactly same as Cascading TemplateTsv.

Tsv(args("input"), ('COUNTRY, 'GDP))
.read
.write(TemplatedTsv(args("output"), "%s", 'COUNTRY))
// it will create a directory for each country under "output" path in Hadoop mode.

Upvotes: 6

Sasha O
Sasha O

Reputation: 3749

This suggestion on the Cascading User group suggests to use Cascading TemplateTap. Not sure how to connect this to Scalding though.

Upvotes: 0

samthebest
samthebest

Reputation: 31513

Use MultipleOutputFormat and extrapolate from these other SO questions to write a custom output class using the output format: Create Scalding Source like TextLine that combines multiple files into single mappers, Compress Output Scalding / Cascading TsvCompressed

Upvotes: 0

Related Questions