CODEWITHSUNDEEP
hadoopmapreduceapache-pig
Sachin Sabbarwal
Sachin Sabbarwal

Reputation: 143

Is there any Pig map task completion hook?

I have a piece of code that I want to run at the end of each of map tasks spawned by pig to perform my job. In other words, I need to do some task just before my map task is exiting. Here is what my research yielded:

  1. We could call PigProgressNotificationListener.jobFinishedNotification() but this method is called on completion of whole job not on completion of every (internal) mapper task.

  2. Finish method in UDF: called at the end of UDF, doesn't meet my requirement.

I am a beginner in MR world.

Upvotes: 0

Views: 75

Answers (1)

Shatu
Shatu

Reputation: 1839

In Hadoop's implementation of MapReduce, there are setup and cleanup functions that are respectively called at the start and end of each of the mappers, and which the developer can override to get the desired functionality.

So, if your Pig script is not that complicated to express as a series of MapReduce programs, you can exploit these functions.

I'm sure that Pig is advanced enough to support such functionality as well. So, just look-up for the Pig equivalent of these functions.

Upvotes: 0

Related Questions