CODEWITHSUNDEEP
jsonhadoopmapreduceapache-pigelephantbird
kskp
kskp

Reputation: 702

Pig Job hangs when using Elephant Bird jsonLoader

This is in continuation to this

My new pig script is:

register /usr/hdp/current/pig-client/lib/piggybank.jar
register /opt/elephantbird-jars/elephant-bird-core-4.5.jar
register /opt/elephantbird-jars/elephant-bird-hadoop-compat-4.5.jar
register /opt/elephantbird-jars/elephant-bird-pig-4.5.jar
register /opt/elephantbird-jars/json-simple-1.1.1.jar

data_input = LOAD 'local/path/for/hdfs/files' USING com.twitter.elephantbird.pig.load.JsonLoader() AS (json:map[]);

x = FOREACH data_input GENERATE json#'actor__id' AS actor_id,    json#'actor__image__url' AS actor_image_url, json#'actor__displayName' AS actor_displayname, json#'actor__verification__adHocVerified' AS actor_verification, json#'actor__url' AS actor_url;

STORE x INTO '/tmp/user_posts' USING JsonStorage();

This code works in local mode: pig -x local user_posts.pig

But it fails in mapreduce mode: pig -x mapreduce user_posts.pig

I moved the jars to all the data nodes in the exact same location. I am not sure where else to check. Can someone please give some pointers?

Upvotes: 0

Views: 97

Answers (2)

kskp
kskp

Reputation: 702

It was some issue with my machine and not any problem with pig. I rebooted the machine and everything went well.

Upvotes: 0

ℕʘʘḆḽḘ
ℕʘʘḆḽḘ

Reputation: 19375

you are missing the ; everywhere

REGISTER '/me/home/elephant-bird-core-4.12.jar';
REGISTER '/me/home/elephant-bird-pig-4.12.jar';
REGISTER '/me/home/elephant-bird-hadoop-compat-4.12.jar';

Upvotes: 0

Related Questions