Searching array of objects in Elasticsearch

Question

I store survey responses in my ES, so what happens is that the people use the mobile app to fill survey forms those form responses are sent to the backend and then stored in the ES like this, this is how answers to the questions in a form are store

answers {
  "uKeCywV4SAgD8YGReSkn" : {
              "_id" : "b465de5e-5468-40fe-8a8d-fc02083",
              "responseVersionNumber" : 1,
              "options" : [
                {
                  "id" : "32700fb5-51d2-4617-b65f-831b83c88080"
                },
                {
                  "id" : "32700fb5-51d2-4617-b65f-831b83c87777"
                }
}

the answers doc contains the answers for each question under its question ID. The string that you see below is the question ID, in the same way all the other answers are stored in ES under their respective question IDs in their respective for doc for that response.

Now I want to search over these answers, the above answer response is of mcq multi-select type, as you can see the text field (that contains) the answer values (i.e. hobbies of the user Gardening and Sports) and the options id array both can be used for this, I can either send the options id array from my web app to ES or the text values (or option labels as we call them) and also using the questions IDs so for example let say I have a question with id q1 so I want my query to say that give me the docs where question q1 has both answers a1 and a2

I have been wanting to explore the options id approach this is a sample query that I build for it.

 {
    "_source": {"includes": ["answers.uKeCywV4SAgD8YGReSkn.text", "answers.uKeCywV4SAgD8YGReSkn.questionTypeCode"]},
    "query": {
        "bool": {
            "filter": [{
                "bool": {
                    "must": [{
                        "match": {
                            "answers.uKeCywV4SAgD8YGReSkn.options.id": "32700fb5-51d2-4617-b65f-831b83c88080"
                        }
                    },
                        {
                        "match": {
                            "answers.uKeCywV4SAgD8YGReSkn.options.id": "32700fb5-51d2-4617-b65f-831b83c87777"
                        }
                    }
                    ]
                }
            }, {
                "match": {
                    "formId": "hHhcLKh9st1vFr8nDY4a"
                }
            }, {
                "range": {
                    "_updated_at": {
                        "from": "1970-01-01T00:00:00.000+0000",
                        "include_lower": true,
                        "include_upper": true,
                        "to": null
                    }
                }
            }]
        }
    }
 }

I wanted to know if there are any drawbacks for going with this approach instead of searching the text field using both the options labels

According to the mapping this is how both the fields are set

"questionId": {
   "type": "keyword",
   "index": false
},

"text": {
    "type": "text"
},

"options": {
    "enabled": false
},

Answer 1

Your question was a bit hard to follow... but it may just be too early in the morning for me :) I don't quite understand what the "options IDs" field is supposed to be doing here. Is your answer being used for multiple questions? Or does your answer have multiple choices? If that's the case, why can a user also reply with text? Also, why put all the fields two levels deep into the document? Why not just store them flat? It seems naming a field as the questionID can be troublesome.

Although I'm not too sure what you're asking here, I can suggest some things that will probably help.

I've been playing around with Elasticsearch recently, specifically with arrays of nested objects. What you might be looking for is the nested datatype. I highly suggest you read the documentation for it.

In short, it will allow you to store your "options IDs" as nested objects that are queryable with their own nested query. This is the easiest way to accomplish searches on nested objects.

First, you need to specify your mapping to tell Elasticsearch that "options" is a nested datatype:

PUT INDEX_NAME
{
  "mappings": {
    "DOCUMENT_TYPE": {
      "properties": {
        "options": {"type": "nested"}
      }
    }
  }
}

Then, after you index your documents, you can use a nested query to query the field. For example, if you want all answers for options ID "ID_123":

GET INDEX_NAME/_search
{
  "query": {
    "nested": {
      "path": "options",
      "query": {
        "match": {
          "options.id.keyword": "ID_123"
        }
      }
    }
  }
}

You can also "bool" the nested query to add in more parameters, such as if you want every answer for an options ID, that has also been updated in the past day.

I hope this helped in some way. If not, try to describe what you are attempting to do with a bit more clarity :)