CODEWITHSUNDEEP
elasticsearchapache-nifi
papelr
papelr

Reputation: 438

NiFi / ElasticSearch: Date Failed to Parse

I'm using NiFi to put some data into ElasticSearch. I created the following ES index:

curl -XPUT http://localhost:9200/test_index -H 'Content-Type: application/json' -d'
{
"index_patterns" : ["test_index"],
"mappings" : {
    "doc"  : {
        "properties" : {
            "example1" :  { "type": "text" },
            "example2" :     { "type": "text" },
            "Date" :         
                { "type": "date",
                  "format": "yyyy-MM-dd HH:mm:ss" }
            }
        }
    }
}'

This loads into ES, no problem. But, the Date object coming through my NiFi flow is this:

"Date" : "Fri, 09 Nov 2018 18:30:49 GMT"

And my PutElasticsearchHttp processor gives me an error:

ERROR [Timer-Driven Process Thread-1] o.a.n.p.e.PutElasticsearchHttp 
PutElasticsearchHttp[id=55b31b9a-3319-124c-4ede-0d3e8c02e9cb] Failed to 
process StandardFlowFileRecord[uuid=5b6d8fc7-0dfd-435d-8a5c- 
adb4cdb08495,claim=StandardContentClaim 
[resourceClaim=StandardResourceClaim[id=1541788260542-269, 
container=default, section=269], offset=176034, 
length=187],offset=0,name=ba74c484-9719-418f-a665- 
7b7d492b40a4,size=187] due to failed to parse [Date], transferring to 
failure

Do I need to change the date format? (Perhaps through UpdateAttribute?)

Upvotes: 0

Views: 1559

Answers (1)

ibexit
ibexit

Reputation: 3667

yes, you need to format the date in nifi to a more common like https://en.wikipedia.org/wiki/ISO_8601 or set a custom date format in your es mapping.

please have a look on patterns described here: https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html

if you need to know how define a custom date mapping in es, have a look on this: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html

Upvotes: 1

Related Questions