CODEWITHSUNDEEP
elasticsearch
MarkII
MarkII

Reputation: 892

Pipeline to delete nested field Elasticsearch mapping

Have an Elasticsearch mapping similar to the one below and I'm trying to update it using the re-index API. I've learned how to use of pipelines to do various things such remove fields or change types, however nothing on removing fields from nested types. For instance, in the descriptions field how would I setup a pipeline to remove the badfield?

{
    "mappings": {
        "all": {
            "_all": {
                "enabled": false
            },
            "dynamic": "strict",
            "properties": {
                "address": {
                    "type": "text"
                },
                "businessName": {
                    "type": "text"
                },
                "descriptions": {
                    "type": "nested",
                    "properties": {
                        "dateSeen": {
                            "type": "date",
                            "format": "date_time"
                        },
                        "source": {
                            "type": "text",
                        },
                        "value": {
                            "type": "text"
                        },
                        "badfield": {
                            "type": "text"
                        }
                    }
                },
                "dateAdded": {
                    "type": "date",
                    "format": "date_time||date_time_no_millis"
                }
            }
        }
    }
}

Documentation on re-indexing

Documentation on removing a field using the removing fields

Using ES 6 btw.

I setup a processor script based on a comment and running into issues where the field is null even though its plainly there.

{
    "processors": [{
            "script": {
                "source": """
                if (ctx._source.descriptions != null) { 
                    for(item in ctx._source.descriptions) { 
                         item.remove('badfield'); 
                    } 
                }
                """
            }
        }
    ]
}

EDIT: removing _source from the script was the issue, which means I don't fully understand its usage but was able to create a nested field removal script.

Upvotes: 0

Views: 1369

Answers (1)

Andrey Borisko
Andrey Borisko

Reputation: 4609

I answered similar question earlier. The logic used there to remove nested could be moved to script processor.

Maybe documentation needs to clarify nested use case but if you just give it a try, it will work (tested on ES 7.5) If you want to deeper understand what happens i guess you need to check the source code

PUT src
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "name": {
        "type": "keyword"
      },
      "nestedField": {
        "type": "nested",
        "properties": {
          "field1": {
            "type": "boolean"
          },
          "field2": {
            "type": "boolean"
          }
        }
      }
    }
  }
}

PUT dst
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "name": {
        "type": "keyword"
      },
      "nestedField": {
        "type": "nested",
        "properties": {
          "field2": {
            "type": "boolean"
          }
        }
      }
    }
  }
}

POST src/_doc
{
  "name": "name1",
  "nestedField": {
    "field1": true,
    "field2": false
  }
}

POST src/_doc
{
  "name": "name2",
  "nestedField": {
    "field1": false,
    "field2": true
  }
}

GET src
GET src/_search
GET dst/_search
GET dst

PUT _ingest/pipeline/test_pipeline
{
  "processors": [
    {
      "remove": {
        "field": "nestedField.field1"
      }
    }
  ]
}

POST _reindex
{
  "source": {
    "index": "src"
  },
  "dest": {
    "index": "dst",
    "pipeline": "test_pipeline"
  }
}

GET dst/_search
GET dst

#DELETE src
#DELETE dst

Upvotes: 2

Related Questions