CODEWITHSUNDEEP
jsonhadoopapache-pig
Eli
Eli

Reputation: 38909

Parse Complex JSON String in Pig

I want to parse a string of complex JSON in Pig. Specifically, I want Pig to understand my JSON array as a bag instead of as a single chararray. When using JsonLoader, I can do this easily by specifying the schema, as in this question. Is there any way to either have Pig figure out my schema for me, or to specify it when Pig is parsing a string? I've been using JsonStringToMap, but can't find a way to specify Schema, or to have it properly understand my JSON array is an array and not a single chararray.

Upvotes: 3

Views: 2050

Answers (1)

Eli
Eli

Reputation: 38909

I wound up using JsonTupleMap() in Mozilla's Akela library for pig. It accomplishes exactly what I want by parsing all of my JSON even when it's complex, and doing this even when I don't provide a schema. If you run into the same problem as me, use that.

Example usage:

REGISTER '/path/to/akela-0.5-SNAPSHOT.jar';
DEFINE JsonTupleMap com.mozilla.pig.eval.json.JsonTupleMap();
loaded = LOAD '$INPUT' AS (json_string:chararray, ...);
jsonified = FOREACH loaded GENERATE JsonTupleMap(json_string) AS json:map[], ...;
some_generate = FOREACH jsonified GENERATE json#'key'#'sub_key';

Upvotes: 4

Related Questions