CODEWITHSUNDEEP
hadoop
user1571324
user1571324

Reputation: 41

How do we pass objects of some custom class as a parameter to mapper in mapReduce programs??

How do we pass objects of some custom class as a parameter to mapper in mapReduce programs?? JobConf has 'set' methods for boolean, string, int and long. What if I want to pass a Document object as a parameter to my mapper? Can any one help me out?

Upvotes: 3

Views: 1860

Answers (2)

Chris White
Chris White

Reputation: 30089

If your object implements Writable, you can serialize it to a Byte array, base64 encode the byte array and then save off the resultant string to the configuration. To decode do the opposite.

Of course, i wouldn't recommend this if your object has a very large footprint - in this case you're better off serializing it to a file in HDFS and using the distributed cache.

Upvotes: 1

Thomas Jungblut
Thomas Jungblut

Reputation: 20969

I have given a tip to someone who wanted a whole map to pass to a mapper.

Hadoop: How to save Map object in configuration

The idea is the same, you have to serialize your object into a string and put it into the configuration. JSON works very well, because the configuration is serialized as XML, thus having no problem while deserializing.

Upvotes: 2

Related Questions