CODEWITHSUNDEEP
hiveapache-nifi
irrelevantUser
irrelevantUser

Reputation: 1322

Nifi PutHive3Streaming - writing to paritioned tables

I am using Nifi 1.7.1 to write to a partitioned hive table. Although the data is streamed successfully, I see several messages in the hive metastore log:

2018-10-29T17:09:40,682 ERROR [pool-10-thread-198]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) - AlreadyExistsException(message:Partition already exists: Partition(values:[2018, 3, 28], dbName:default, tableName:myTable, createTime:0, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:type, type:string, comment:null), FieldSchema(name:id, type:string, comment:null), FieldSchema(name:referenced_event_id, type:string, comment:null), FieldSchema(name:happened, type:string, comment:null), FieldSchema(name:processed, type:string, comment:null), FieldSchema(name:tracking_id, type:string, comment:null), FieldSchema(name:source_attributes, type:struct<id:string,origin:string,data:map<string,string>,external_data:map<string,string>>, comment:null), FieldSchema(name:event_data, type:struct<service:struct<name:string,version:string>,result:struct<mno:string,mvno:string,mcc:string,mnc:string,country:string>>, comment:null)], location:hdfs://node-master:8020/user/hive/warehouse/myTable/year=2018/month=3/day=28, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, numBuckets:6, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{serialization.format=1}), bucketCols:[tracking_id], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), parameters:null, catName:hive))

I have tried this with:

"hive3-stream-part-vals": "${year},${month},${day}",
"hive3-stream-autocreate-partition": "false",

and also with

"hive3-stream-autocreate-partition": "true",

Does anyone have a clear idea on why these errors are being logged?

Upvotes: 0

Views: 505

Answers (1)

mattyb
mattyb

Reputation: 12083

I think you're running into https://issues.apache.org/jira/browse/HIVE-18931. What is your setting for the processor's Max Concurrent Tasks property? If it is greater than 1, can you try setting it to 1 and see if you still get the message? If it is 1, are multiple clients (NiFi, beeline, etc.) trying to write to that table at the same time?

Upvotes: 2

Related Questions