CODEWITHSUNDEEP
javahadoopapache-pighue
User456898
User456898

Reputation: 5724

Apache Pig - Create unique output folder using UDF

I am using Apache Pig from Hue to perform ETL operations on files using the script etl-op.pig. The output is stored into the specified folder in HDFS using the following line:

STORE outval INTO '/user/root/Pig-Output

However next time when the script is run, it says the output folder already exists and doesn't create a separate folder.

Is there any way to create a Java UDF in Pig using Hue so that a unique identifier can be generated and appended to the 'Pig-Output' folder name present in the script ?

Upvotes: 2

Views: 440

Answers (1)

kecso
kecso

Reputation: 2485

You can do it without UDF: Define a variable like the current unix timestamp:

%default TS `date  +%s`

And than use it as e.g. a postfix of your folder:

STORE outval INTO '/user/root/Pig-Output_$TS' ...

Upvotes: 1

Related Questions