CODEWITHSUNDEEP
elasticsearchlogstashkibana
suren
suren

Reputation: 8786

How to force Elastic to keep more decimals from a float

I have some coordinates that I pass to Elasticsearch from Logstash, but Elastic keeps only 3 decimals, so coordinate wise, I completely lose the location.

When I send the data from Logstash, I can see it got the right value:

{
          "nasistencias" => 1,
         "tiempo_demora" => "15",
                  "path" => "/home/elk/data/visits.csv",
                 "menor" => "2",
               "message" => "5,15,Parets del Vallès,76,0,8150,41.565505,2.234999575,LARINGITIS AGUDA,11/3/17 4:20,1,38,1,2,POINT(2.2349995750000695 41.565505000000044)",
          "id_poblacion" => 76,
            "@timestamp" => 2017-03-11T04:20:00.000Z,
             "poblacion" => "Parets del Vallès",
            "edad_valor" => 0,
             "patologia" => "LARINGITIS AGUDA",
                  "host" => "elk",
              "@version" => "1",
    "Geopoint_corregido" => "POINT(2.2349995750000695 41.565505000000044)",
               "id_tipo" => 1,
                "estado" => "5",
                    "cp" => 8150,
              "location" => {
        "lon" => 2.234999575,               <- HERE
        "lat" => 41.565505                  <- AND HERE
    },
           "id_personal" => 38,
                 "Fecha" => "11/3/17 4:20"
}

But then, I get it on Kibana as follows:

enter image description here

I do the conversion as follows:

mutate {
    convert => { "longitud_corregida" => "float" }
    convert => { "latitude_corregida" => "float" }
}
mutate {
  rename => {
      "longitud_corregida" => "[location][lon]"
      "latitude_corregida" => "[location][lat]"
  }
}

How can I keep all the decimals? With geolocation, one decimal can return the wrong city.

Another question (related)

I add the data to the csv document as follows:

# echo "5,15,Parets del Vallès,76,0,8150,"41.565505","2.234999575",LARINGITIS AGUDA,11/3/17 4:20,1,38,1,2,POINT(2.2349995750000695 41.565505000000044)" >> data/visits.csv

But in the original document, instead of dots there are comas for the coordinates. like this:

# echo "5,15,Parets del Vallès,76,0,8150,"41,565505","2,234999575",LARINGITIS AGUDA,11/3/17 4:20,1,38,1,2,POINT(2.2349995750000695 41.565505000000044)" >> data/visits.csv

But the problem was that it was getting the coma as field separator, and all the data was being sent to Elasticsearch wrong. Like here:

enter image description here

Here, the latitude was 41,565505, but that coma made it understand 41 as latitude, and 565505 as longitude. I changed the coma by dot, and am not sure if float understands comas and dots, or just comas. My question is, did I do wrong changing the coma by dot? Is there a better way to correct this?

Upvotes: 0

Views: 669

Answers (1)

ibexit
ibexit

Reputation: 3667

Create a GEO-Point mapping for the lat/lon fields. This will lead to a more precise and internally optimized storage in ES and allow you more sophisticated GEO-Queries.

Please keep in mind, that you'll need to reindex the data as mapping changes are not possible afterwards (if there are already docs present having the fields to change)

Zero downtime approach:

  1. Create a new index with a optimized mapping (derive it from the current, and make your changes manually)
  2. Reindex the data (at least some docs for verification)
  3. Empty the new index again
  4. Change the logstash destination to the new index (consider using aliases)
  5. Reindex the old data into the new index

Upvotes: 1

Related Questions