CODEWITHSUNDEEP
javaelasticsearchgsondatetime-formatdatetime-parsing
Gabriel
Gabriel

Reputation: 624

Malformed Date Error Indexing Data Into Elasticsearch

I'm attempting to index data in elasticsearch that includes dates with time zones.

My date mapping is:

"date": {
    "type": "date",
    "format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
}

I use the following code to index the data:

client.prepareIndex(INDEX, TYPE, id))
        .setSource(gson.toJson(object), XContentType.JSON)
        .setRefreshPolicy(RefreshPolicy.IMMEDIATE)
        .get();

I've created my own ZonedDateTime adapter as follows:

public class ZonedDateTimeAdapter implements JsonSerializer<ZonedDateTime> {

    public JsonElement serialize(ZonedDateTime date, Type typeOfSrc, JsonSerializationContext context) {
        DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
        return new JsonPrimitive(date.format(formatter));
    }

}

The resulting date is like 2005-01-01T13:35:50.596-0500. Which to me seems to match my format of yyyy-MM-dd'T'HH:mm:ss.SSSZ. However I receive an error stating:

Exception in thread "main" MapperParsingException[failed to parse [startDate]]; nested: IllegalArgumentException[Invalid format: "2005-01-01T13:35:50.596-0500" is malformed at ".596-0500"]

What's interesting is that if I change format in ZonedDateTimeAdapter to read yyyy-MM-dd'T'HH:mm:ss.000Z (forcing the second fraction to always be 000) I get a resulting date like 2005-01-01T13:31:06.000-0500. An object with this date is successfully indexed into elasticsearch.

Do you know why 2005-01-01T13:31:06.000-0500 index successfully but 2005-01-01T13:35:50.596-0500 does not? Aren't these the same format?

Upvotes: 2

Views: 880

Answers (2)

Gabriel
Gabriel

Reputation: 624

I'm not sure if my "fix" will be appropriate for anyone seeing this problem. But it was getting worse. I couldn't parse out any dates that ended in 0. So yyyy-MM-dd'T'HH:mm:00 and yyyy-MM-dd'T'HH:00 would fail, but yyyy-MM-dd'T'HH:mm:01 or yyyy-MM-dd'T'HH:01 would be successful. I destroyed my entire elasticsearch database and upgraded to 6.1 and reindexed all of my data. I don't see the issue anymore.

Upvotes: 0

kgf3JfUtW
kgf3JfUtW

Reputation: 14926

Update: I did a quick test on my ES 5.2 and it worked fine.

$curl -XPUT localhost:9200/myindex1 -d '
{"mappings": {"type1": {"properties": {"field1": {
    "type": "date", 
    "format": "yyyy-MM-dd'\''T'\''HH:mm:ss.SSSZ"
}}}}}
'

$curl -XPUT localhost:9200/myindex1/type1/id1 -d '
    { "field1":"2005-01-01T13:35:50.000-0500" }
'

$curl -XPUT localhost:9200/myindex1/type1/id2 -d '
    { "field1":"2005-01-01T13:35:50.596-0500" }
'

To confirm we have the same mapping:

$curl localhost:9200/myindex1/type1/_mapping
{"myindex1":{"mappings":{"type1":{"properties":
    {"field1":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}}}}}}

From this thread on https://discuss.elastic.co

Currently, Elasticsearch is limited to millisecond precision, see this GH issue.

The post is from 2015 but the issue remains open. It looks like nanosecond precision is not yet supported.

Upvotes: 2

Related Questions