CODEWITHSUNDEEP
elasticsearchkibana
Jimmy
Jimmy

Reputation: 12487

Kibana fails to pick up date from elasticsearch when I include the hour and minute

I'm really stuggling to get this specific time format into elasticsearch so I can graph it in Kibana. I cannot change this format. My elasticsearch data and mapping is in this format:

STEP 1: Setup Mapping

PUT http://<>.com:5101/myindex6/_doc/1

{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "HH:mm yyyy-MM-dd"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}

Step 2: Add Data

PUT http://<>.com:5101/myindex6

{
  "test" : [ {
    "data" : "119050300",
    "date" : "10:00 2019-06-03"
  } ]
}

In Kibana it wont find this as a date and wont allow me to map it as one. However, if I remove the time aspect and use the date, and do this instead, it works fine:

Data

{
  "test" : [ {
    "data" : "119050300",
    "date" : "2019-06-03"
  } ]
}

Map

{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "yyyy-MM-dd"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}

Can someone please tell me how to include the time and not have it break, so I can filter on time in kibana.

Upvotes: 7

Views: 259

Answers (5)

Nicolae Maties
Nicolae Maties

Reputation: 2655

This is how I think it will work.

PUT http://<>.com:5101/myindex6/_doc/1

{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "yyyy-MM-dd HH:mm:ss"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}

PUT http://<>.com:5101/myindex6

{
  "test" : [ {
    "data" : "119050300",
    "date" : "2019-06-03T10:00:00Z"
  } ]
}

Upvotes: 0

Anshu
Anshu

Reputation: 1327

You've got it all right, except one single tiny thing. In your mapping you specify the date format as being epoch_millis but in your document the datetime is specified in epoch_second.

So you have two choices:

  1. Either you keep the mapping as it is and multiply all your date times by a factor of 1000
  2. Or you change your mapping with "format": "epoch_second" and keep your data as it is.

In any case, you need to reload your index pattern in Kibana afterwards and it will work. Note that I tested this on Kibana 4.2.

Upvotes: 0

avp
avp

Reputation: 3330

As everyone has already pointed out, your index mapping differs from your data, so you need to update the mapping. I would highly suggest revising Elasticsearch's index mapping guide to get a good understanding of mappings and their effect on the data indexed.

On another note, I saw you are still using mapping types. If you are just getting started with your ES use cases, it will be good to avoid using them as they will be deprecated and eventually removed from ES [source].

Upvotes: 0

Surbhi Harsh
Surbhi Harsh

Reputation: 69

As already pointed in the above answer, the data which you are indexing doesn't match with the mapping created. What you need to do is update your mapping with test field as nested.

{
  "mappings": {
    "properties": {
        "test": {
            "type": "nested",
            "properties": {
                "date": {
                    "type": "date",
                    "format": "HH:mm yyyy-MM-dd"
                },
                "data": {
                    "type": "integer"
                }
            }
        }
    }
}}

And then re-index your data. The date field will appear in Kibana after that.

Upvotes: 1

Nishant
Nishant

Reputation: 7854

There is the difference between the mapping and the structure document that you are indexing. Also the endpoints you are using seems be swapped. Follow the steps below:

1. Create index 
PUT myindex6
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "HH:mm yyyy-MM-dd"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}
2. Index document 
POST myindex6/_doc/1
{
  "data": "119050300",
  "date": "10:00 2019-06-03"
}

Notice the endpoints used to create index and then to index a document. Also notice the structure of document is in line with the mapping. In your case you are indexing a document with a test field which is an array of object with fields data and date. This structure doesn't match to the mapping created in step 1.

Upvotes: 3

Related Questions