CODEWITHSUNDEEP
elasticsearch
Denys
Denys

Reputation: 4557

Add a field with a dynamically mapped value to all documents

There is an index shakespeare with the documents looking like following:

{
        "_index" : "shakespeare",
        "_type" : "line",
        "_id" : "24",
        "_score" : 1.0,
        "_source" : {
          "play_name" : "Henry IV",
          "speech_number" : 1,
          "line_number" : "1.1.22",
          "text_entry" : "Forthwith a power of English shall we levy;",
          "speaker" : "KING HENRY IV",
          "line_id" : 1
        }
      },

I have a file/table/etc., which looks like following

+---------+--------------+
| line_id | is_a_good_line |
+---------+--------------+
|       1 |            1 |
|       2 |            1 |
|       3 |            0 |
|       4 |            0 |
|       5 |            1 |
+---------+--------------+

I want for each document in the shakespeare index have a new field is_a_good_line with a value matching the mapping from the table (based on the line_id). So, for the above example the value of is_a_good_line should be 1 like below:

{
        "_index" : "shakespeare",
        "_type" : "line",
        "_id" : "24",
        "_score" : 1.0,
        "_source" : {
          "play_name" : "Henry IV",
          "speech_number" : 1,
          "line_number" : "1.1.22",
          "text_entry" : "Forthwith a power of English shall we levy;",
          "speaker" : "KING HENRY IV",
          "is_a_good_line" : 1,
          "line_id" : 1
        }
      },

I found a way to simply add a new field to all documents with a static value:

POST shakespeare/_update_by_query
{
  "script": {
    "inline": "ctx._source.is_a_good_line = 0",
    "lang": "painless"
  }
}

How do I add a new field with a value defined by the mapping table above (based on the line_id value)?

Upvotes: 0

Views: 146

Answers (1)

glenacota
glenacota

Reputation: 2547

There are several options, from writing a simple program that parses the file/table and updates the relevant documents accordingly, to defining a Logstash pipeline that does the same job by using the appropriate input/output plugins. For sure, you cannot rely on Elasticsearch only to parse an external file.

If you want to go for the program option, it can be something as easy as a bash script that parses the file and curls _update_by_query with a query like this

{
  "query": {
    "match": { "line_id": <the_line_id_fetched_from_the_file>
  },
  "script": {
    "inline": "ctx._source.is_a_good_line = <the_is_a_good_line_value_from_the_file>",
    "lang": "painless"
  }
}

or you can develop the same program in your favourite language using one of the many official elasticsearch client libraries: https://www.elastic.co/guide/en/elasticsearch/client/index.html

If instead you want to go for the Logstash pipeline option, you may want to have a look to the file input plugin and elasticsearch output plugin (check out the script option in particular), and maybe to the csv filter plugin (depending on the way the data is structured in your file/table).

Upvotes: 1

Related Questions