CODEWITHSUNDEEP
javahadoopmapreducehbasehdfs
sangfroid
sangfroid

Reputation: 3803

Is a single hadoop Mapper object used for more than one call to map()?

So, I have a Mapper that updates an HBase table. In the map() function, I :

1) instantiate an HBaseConfiguration

2) instantiate an HTable

3) call hTable.put() a bunch of times to add rows

4) call hTable.flushCommits() to flush my changes

5) call HConnectionManager.deleteConnection() to kill the connection to HBase

However, this seems inefficient. I would like to instantiate the HBaseConfiguration and the HTable in the constructor for my Mapper class. Then I could have my mapper class implement Closeable, calling hTable.flushCommits() and HConnectionManager.deleteConnection() in the close() method. That way, in each call to map(), I'd be buffering my put() calls, and would be flushing all the changes at once, when close() is called.

However, this is only worthwhile if the Mapper object is re-used for multiple calls to map(). Otherwise, I may as well leave my code alone.

So the main question is : are Mapper objects used for more than one call to map()?

The bonus question is : would the re-written code be more efficient?

Upvotes: 4

Views: 648

Answers (1)

Donald Miner
Donald Miner

Reputation: 39913

What you are looking for is setup and cleanup. setup runs once before map is called a bunch of times, and cleanup is called once after all the maps are called. You override these just like you override map.

Use private member objects for your HBaseConfiguration and HTable. Initialize them in the setup. Do your hTable.put() in your map. Do hTable.flushCommits() and HConnectionManager.deleteConnection() in your cleanup. The only thing you might want to be careful is to flush the commits more than just at the end in the case you are buffering more data than your memory can handle. In which case, you might want to flush every 1000 records or something in the map by keeping track of number of records you've seen.

This will definitely be more efficient! Opening and closing that connection is going to incur a significant amount of overhead.

Check out the documentation for Mapper

Upvotes: 4

Related Questions