CODEWITHSUNDEEP
hadoopstoreapache-pigbigdata
mytwocents
mytwocents

Reputation: 857

Hadoop Pig cannot store to an existing folder

I have created a folder to drop the result file from a Pig process using the Store command. It works the first time, but the second time it compains that the folder already exists. What is the best practice for this situiation? Documentation is sparse on this topic.

My next step will be to rename the folder to the original file name, to reduce the impact of this. Any thoughts?

Upvotes: 3

Views: 6891

Answers (1)

Chris White
Chris White

Reputation: 30089

You can execute fs commands from within Pig, and should be able to delete the directory by issuing a fs -rmr command before running the STORE command:

fs -rmr dir
STORE A into 'dir' using PigStorage();

The only subtly is the fs command doesn't expect quotes around the directory name, whereas the store command does expect quotes around the directory name.

Upvotes: 8

Related Questions