CODEWITHSUNDEEP
apache-pig
MrRaymondLee
MrRaymondLee

Reputation: 536

How are Apache Pig UDFs distributed to data nodes?

There are plenty of documentation about how to write Pig UDFs in the various languages but I haven't found anything on how they are distributed to the data nodes.

Are they done automatically when pig script is invoked? If it makes any difference, I'd be writing UDF in Java.

Upvotes: 0

Views: 87

Answers (2)

hello_abhishek
hello_abhishek

Reputation: 545

Let me make it more clear. Whenever we wite a UDF and the pig is in hdfs mode. Then UDFs, which initially resides in the local or the client side, is carried to the cluster as per the internal architecture of hadoop. Now the UDFs task is performed by the task tracker and it becomes the duty of the job tracker to assign the the UDFs to task tracker, which is near to the data node where the input file resides. Note: Its always the job tracker(component of name node), which actually decides which task tracker should perform the execution of the UDFs.

If the input file is in local file system(local mode), then the UFDs get executed in the local JVM.

Upvotes: 1

hello_abhishek
hello_abhishek

Reputation: 545

The fact is apache pig works in two modes 1) local mode 2) hdfs mode

To answer you question, which belongs to pig running in hdfs mode, we only made sure that the input file that we are loading is present in the hdfs(data node). When the question comes for UDF, this is simply a function that is used to process the input file, just link pig latin language. We are writing UDFs, pig latin via the client side node and thus all the data related to this will be stored in the client side machine. Above all, we have configure the pig so that client can interact with the hdfs to process the required result.

Hope this helps

Upvotes: 0

Related Questions