Elastic Search 8+ painless script filtering issue (no access to params._source via query)

Question

I have an issue with ES filtering, and hope someone share a hint about it.

Mapping:

{
    "mappings": {
        "properties": {
            "some_availabilities": {
                "properties": {
                    "default": {
                        "properties": {
                            "mark": {
                                "properties": {
                                    "somedata1": {
                                        "type": "boolean"
                                    },
                                    "somedata2": {
                                        "type": "boolean"
                                    },
                                    "offline": {
                                        "type": "long"
                                    },
                                    "online": {
                                        "type": "long"
                                    }
                                }
                            },
                            "lily": {
                                "type": "nested",
                                "properties": {
                                    "somedata1": {
                                        "type": "boolean"
                                    },
                                    "somedata2": {
                                        "type": "boolean"
                                    },
                                    "offline": {
                                        "type": "long"
                                    },
                                    "online": {
                                        "type": "long"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

Data:

{
    "some_availabilities": {
        "default": {
            "lily": [],
            "mark": [
                {
                    "online": 1645480000,
                    "offline": 1645481000,
                    "somedata1": false,
                    "somedata2": false
                },
                {
                    "online": 1645482000,
                    "offline": 1645483000,
                    "somedata1": true,
                    "somedata2": false
                },
                {
                    "online": 1645484000,
                    "offline": 1645485000,
                    "somedata1": false,
                    "somedata2": true
                },
                {
                    "online": 1645486000,
                    "offline": 1645487000,
                    "somedata1": true,
                    "somedata2": true
                }
            ]
        }
    }
}

I need to go through all objects inside Mark's array, do some checks and in case some of them is true - script should respond with true value to show doc after after filtering.

ES request:

"query": {
        "bool": {
            "filter": {
                "script": {
                    "script": {
                        "source": "def result=false;long timestamp=params['timestamp'];def baseKey='some_availabilities.default.mark'; if(doc.containsKey(baseKey+ '.online')){def ctxAvailOnline = doc[baseKey + '.online'];for(int n=0;n0){online=doc[baseKey+'.online'][n];offline=doc[baseKey+'.offline'][n];field1=doc[baseKey+'.somedata1'][n];field2=doc[baseKey+'.somedata2'][n];if(online<=timestamp&&offline>=timestamp&&field1&&!field2){result=true; return result}}}} ",
                        "lang": "painless",
                        "params": {
                            "timestamp": 1645482000
                        }
                    }
                }
            }
        }
    },

If we check data, for current params timestamp value it's a third element of 'mark' property and document should be shown after filtering. But, actually, ES doesn't save it as real array of objects, it uses some optimization mechanism and it's not guaranteed that FE in doc[baseKey+'.somedata2'][2] I'll have 'true' value. ES keeps data like separated optimised array and inside some_availabilities.default.mark.somedata2 array, it could have [true,true,false,false] and referring to second element, doesn't mean it's the same second that's been defined during _doc creation.

Before version 6, I used params._source, for getting exact document, that's been defined and it helped, but now - it's not allowed to get data from params._source inside filtering/inline script scope.

I tried to use "nested" type for 'default' and 'mark/lily' properties, but it didn't help me with my goal.

I checked script_fields, using them I can have needed data, because it allows to use params._source, but this field couldn't be used for filtering condition because of many reasons.

Does someone have an idea, how to go through the nested object like mine, and check some conditions inside one level like "mark's->fields" and keep the order of it. Or, what the another way of handling the case like this?

Elastic Search 8+ painless script filtering issue (no access to params._source via query)

Answers (1)

There are some missing point in your mapping and query. I'm sharing the updated mapping and the script query please check.

Related Questions