CODEWITHSUNDEEP
hadoopapache-pighdfs
Uno
Uno

Reputation: 543

Hadoop Load and Store

When I am trying to run a Pig script which has two "store" to the same file this way

store Alert_Message_Count into 'out';
store Warning_Message_Count into 'out';

It hangs, I mean it does not proceed after showing 50% done.

Is this wrong? Cant we store both the results in the same file(folder)?

Upvotes: 1

Views: 1954

Answers (2)

pyfunc
pyfunc

Reputation: 66709

HDFS does not have append mode. So in most cases where you are running map-reduce programs, the output file is opened once, data is written and then closed. Assuming this approach you can not write data simultaneously onto the same file.

Try writing to separate files and check if the map-red programs do not hang. If they still do, then there are some other issues.

You can obtain the result and map-reduce logs to analyze what went wrong.

[Edit:]

You can not write to the same file or append to an existing file. The HDFS Append feature is a work in progress.

To work on this you can do two things:

1) If you have the same schema content in both Alert_Message_Count and Warning_Message_Count, you could use union as suggested by Chris.

2) Do post processing when the schema is not the same. That is write a map reduce program to merge the two separate outputs into one.

Upvotes: 1

Chris White
Chris White

Reputation: 30089

Normally Hadoop MapReduce won't allow you to save job output to a folder that already exists, so i would guess that this isn't possible either (seeing as Pig translates the commands into a series of M/R steps) - but i would expect some form of error message rather than it just to hang.

If you open the cluster job tracker, and look at the logs for the task, does the log yield anything of note which can help diagnose this further?

Might also be worth checking with the pig mailing lists (if you haven't already)

If you want to append one dataset to another, use the union keyword:

grunt> All_Count = UNION Alert_Message_Count, Warning_Message_Count;
grunt> store All_Count into 'out';

Upvotes: 0

Related Questions