CODEWITHSUNDEEP
elasticsearch
Muhammad Ali
Muhammad Ali

Reputation: 732

Ingest pipeline with update_by_query on failure causes version_conflict_engine_exception

Situation: I set the "on_failure" statement to an existing pipeline like follows (for the whole pipeline):

"on_failure" : [
  {
    "set" : {
      "field" : "_index",
      "value" : "failed-{{ _index }}"
    }
  },
  {
    "set" : {
      "field" : "caused_by",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }
]

Problem: The pipeline is used with the update_by_query API to enrich documents. In case of a failure, the pipeline does not forward the failed document to the failed index ie. failed-my_index. This results due to the following error message:

{
  "took": 276,
  "timed_out": false,
  "total": 1,
  "updated": 0,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 1,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures": [
    {
      "index": "failed-my_index",
      "type": "_doc",
      "id": "1002109",
      "cause": {
        "type": "version_conflict_engine_exception",
        "reason": "[1002109]: version conflict, required seqNo [37265539], primary term [20]. but no document was found",
        "index_uuid": "JMepgCegQamU8WDqQuFJ3Q",
        "shard": "0",
        "index": "failed-my_index"
      },
      "status": 409
    }
  ]
}

This is true, because the document "1002109" doesn't exist in the failed index, because it didn't fail when it was indexed originally. On update, the pipeline fails, and the document needs to go into the failed index. This is not happening because of the version conflict error above.

What am I doing wrong here?

Any help is appreciated

Upvotes: 0

Views: 630

Answers (1)

Val
Val

Reputation: 217324

As stated in the official documentation about the _update_by_query endpoint:

This API only enables you to modify the source of matching documents, you cannot move them.

This information is also available in the scripting documentation regarding what can be done in the update by query context as we can see that most of the ctx._* fields are read-only:

  • ctx['_routing'] (String, read-only) : The value used to select a shard for document storage.
  • ctx['_index'] (String, read-only): The name of the index.
  • ctx['_type'] (String, read-only): The type of document within an index.
  • ctx['_id'] (int, read-only): The unique document id.
  • ctx['_version'] (int, read-only): The current version of the document.

So, it's not possible to change the index of a document using update by query. You can, however, flag the documents and then use a combination of reindex + delete-by-query to move those flagged documents to your failed index.

Upvotes: 1

Related Questions