CODEWITHSUNDEEP
elasticsearchlogstashkibana
Lorenz Power
Lorenz Power

Reputation: 67

Logstash - how to change field types

Hello guys I am new to Elasticsearch and I have a table called 'american_football_sacks_against_stats'. It consists of three columns.

id sacks_against_total sacks_against_yards
1 12 5
2 15 3
... ... ...

The problem is that sacks_against_total and sacks_against_yards aren't 'imported' as integers/longs/floats whatsoever but as a text field and a keyword field. How can I convert them into numbers?

I tried this but its not working:

mutate {
        convert => {
             "id" => "integer"
             "sacks_against_total" => "integer"
             "sacks_against_yards" => "integer"
        } 
    }

This is my logstash.conf file:

input {
    jdbc {
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/sportsdb"
        jdbc_user => "user"
        jdbc_password => "password"
        jdbc_driver_class => "org.postgresql.Driver"
        schedule => "*/5 * * * *" 
        statement => "SELECT * FROM public.american_football_sacks_against_stats"
        jdbc_paging_enabled => "true"
        jdbc_page_size => "300"
    }
}

filter {
    mutate {
        convert => {
             "id" => "integer"
             "sacks_against_total" => "integer"
             "sacks_against_yards" => "integer"
        } 
    }
}

output {
    stdout { codec => "json" }
    elasticsearch {
        hosts => "http://localhost:9200"
        index => "sportsdb"
        doc_as_upsert => true #
    }
}

Upvotes: 1

Views: 4346

Answers (2)

Lorenz Power
Lorenz Power

Reputation: 67

This is the solution I was looking for: https://linuxhint.com/elasticsearch-reindex-change-field-type/

To start you create a input from your index and change the types.

PUT _ingest/pipeline/convert_pipeline
{
  “description”: “converts the field sacks_against_total,sacks_against_yards fields to a long from string”,
  "processors" : [
    {
      "convert" : {
        "field" : "sacks_against_yards",
        "type": "long"
      }
    }, 
    {
      "convert" : {
        "field" : "sacks_against_total",
        "type": "long"
      }
    }
  ]
}

Or in cURL:

curl -XPUT "http://localhost:9200/_ingest/pipeline/convert_pipeline" -H 'Content-Type: application/json' -d'{  "description": "converts the sacks_against_total field to a long from string",  "processors" : [    {      "convert" : {        "field" : "sacks_against_total",        "type": "long"      }    }, {"convert" : {        "field" : "sacks_against_yards",        "type": "long"     } }  ]}'

And then reindexing the index into another one

POST _reindex
{
  “source”: {
    "index": "american_football_sacks_against_stats"
  },
  "dest": {
    "index": "american_football_sacks_against_stats_withLong",
    "pipeline": "convert_pipeline"
  }
}

Or in cURL:

curl -XPOST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' -d'{  "source": {    "index": "sportsdb"  },  "dest": {    "index": "sportsdb_finish",    "pipeline": "convert_pipeline"  }}'

Upvotes: 1

Todor Sidorov
Todor Sidorov

Reputation: 1

Elasticsearch does not provide the functionality to change types for existing fields.

You can read here for some options: https://medium.com/@max.borysov/change-field-type-in-elasticsearch-index-2d11bb366517

Upvotes: 0

Related Questions