CODEWITHSUNDEEP
javajsonhadoophiveelastic-map-reduce
saschor
saschor

Reputation: 319

Hive / ElasticMapreduce: How bring JsonSerDe to ignore malformed JSON?

I'm fairly new to Hive and ElasticMapreduce and currently im stuck to a particular problem. When running a Hive statement on a table with billions of lines of JSON objects, the MapReduce job crashes as soon as only one of those lines is invalid / malformed JSON.

Exception:

java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"ip":"39488130","cdate":"2012-08-09","cdate_ts":"2012-08-09 17:06:41","country":"SA","city":"Riyadh","mid":"6666276582211270592","osversion":"5.1.
1
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:441)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:377)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"ip":"39488130","cdate":"2012-08-09","cdate_ts":"2012-08-09 17:06:41","country":"SA","city":"Riyadh","mid":"6666276582211270592","osversion":"5.1.1
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string near
at com.google.gson.Streams.parse(Streams.java:51)
at com.google.gson.JsonParser.parse(JsonParser.java:83)
at com.google.gson.JsonParser.parse(JsonParser.java:58)
at com.google.gson.JsonParser.parse(JsonParser.java:44)
at com.amazon.elasticmapreduce.JsonSerde.deserialize(Unknown Source)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:510)
... 9 more
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated string near
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1110)
at com.google.gson.stream.JsonReader.nextString(JsonReader.java:967)
at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:802)
at com.google.gson.stream.JsonReader.objectValue(JsonReader.java:782)
at com.google.gson.stream.JsonReader.quickPeek(JsonReader.java:377)
at com.google.gson.stream.JsonReader.peek(JsonReader.java:340)
at com.google.gson.Streams.parseRecursive(Streams.java:60)
at com.google.gson.Streams.parseRecursive(Streams.java:83)
at com.google.gson.Streams.parse(Streams.java:40)
... 14 more

I create my tables like this:

CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
column1 string,
column2 string
)
PARTITIONED BY (year string, month string)
ROW FORMAT SERDE 'com.amazon.elasticmapreduce.JsonSerde'
WITH SERDEPROPERTIES ('paths'='c1, c2')
LOCATION 's3://mybucket/table1';

What can I do to prevent the crashing? Ignoring malformed JSON objects / strings would be fine, since its one out of billions which is malformed.

Thanks for your help in advance. Best, Sascha

Upvotes: 4

Views: 4917

Answers (3)

mountrix
mountrix

Reputation: 1243

By changing the class used in the row format and adding the 'malformed' property, you can make your table creation working on malformed JSONs:

ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ("ignore.malformed.json" = "true")
LOCATION ...

Include the JAR using the 'hive.aux.jars.path' property in 'hive-site.xml' or the 'ADD JAR' Hive instruction. You can find the JAR here, or compile it from this source.

Upvotes: 4

saschor
saschor

Reputation: 319

the JsonSerDe from Apache seems to ignore malformed JSON strings... http://code.google.com/p/hive-json-serde/

Upvotes: 0

KSReddy
KSReddy

Reputation: 55

Basically,the above error is occurring because JSON invalid string. Try to resolve this issue.

Othere wise,to avoid crashing your app,catch that exception in try catch block and proced further.So that your app will not crash.

Upvotes: -2

Related Questions