Reputation: 15
Search with more than one parameter over more than one field in elastic search
I only want to return this course if 'Grade' = 'G6' and Type = 'Open' are a match in the SAME audience tag, they must exist in the SAME tag to return this course. Currently this course is returned if it finds G6 and OPEN is DIFFERENT audiences which is not what I want. This is not correct and i am getting incorrect data back, I need to query to apply in each audience and only return data if it is true in the same audience
here is my json:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 71,
"max_score": 3.3118114,
"hits": [
{
"_index": "courses",
"_type": "course",
"_id": "LBTBWdzyRw-jgiiYssjv8A",
"_score": 3.3118114,
"_source": {
"id": "LBTBWdzyRw-jgiiYssjv8A",
"title": "1503 regression testing",
"shortDescription": "asdf",
"description": "asdf",
"learningOutcomes": "",
"modules": [],
"learningProvider": {
"id": "ig2-zIY_QkSpMC4O0Lm0hw",
"name": null,
"termsAndConditions": [],
"cancellationPolicies": []
},
"audiences": [
{
"id": "VfDpsS_5SXi8iZubzTkUBQ",
"name": "comm",
"areasOfWork": [
"Communications"
],
"departments": [],
"grades": [
"G6"
],
"interests": [],
"requiredBy": null,
"frequency": null,
"type": "OPEN",
"eventId": null
},
{
"id": "eZPPPqTqRdiDAE3xCPlJMQ",
"name": "analysis",
"areasOfWork": [
"Analysis"
],
"departments": [],
"grades": [
"G6"
],
"interests": [],
"requiredBy": null,
"frequency": null,
"type": "REQUIRED",
"eventId": null
}
],
"preparation": "",
"owner": {
"scope": "LOCAL",
"organisationalUnit": "co",
"profession": 63,
"supplier": ""
},
"visibility": "PUBLIC",
"status": "Published",
"topicId": ""
}
}
]
}
}
My ES Code:
BoolQueryBuilder boolQuery = boolQuery();
boolQuery.should(QueryBuilders.matchQuery("audiences.departments.keyword", department));
boolQuery.should(QueryBuilders.matchQuery("audiences.areasOfWork.keyword", areaOfWork));
boolQuery.should(QueryBuilders.matchQuery("audiences.interests.keyword", interest));
BoolQueryBuilder filterQuery = boolQuery();
filterQuery.must(QueryBuilders.matchQuery("audiences.grades.keyword", "G6"));
filterQuery.must(QueryBuilders.matchQuery("audiences.type", "OPEN"));
Here is index mapping:
{
"media": {
"aliases": {}
},
"courses": {
"aliases": {}
},
"feedback": {
"aliases": {}
},
"learning-providers": {
"aliases": {}
},
"resources": {
"aliases": {}
},
"courses-0.4.0": {
"aliases": {}
},
".security-6": {
"aliases": {
".security": {}
}
},
"payments": {
"aliases": {}
}
}
Upvotes: 1
Views: 251
Answers (1)
Reputation:
Since you want your query to apply in each audience and only return data if it is true in the same audience, you need to specify nested datatype for audiences field otherwise ElasticSearch stores it in form of Objects and it doesnt have concept of nested objects because of which Elasticsearch flattens object hierarchies into a simple list of field names and values.You can refer this for more detail https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html
Taking your example suppose this was your document :
"audiences": [
{
"id": "1",
"field": "comm"
},
{
"id": "2",
"field": "arts"
}
]
Elasticsearch flattens in the form of :
{
"audiences.id":[1,2],
"audiences.field":[comm,arts]
}
Now here if you search query says that audience must have id:1 and field:arts then also above document will get matched.
So, in order to avoid this such type of objects should be defined as nested object. ElasticSearch will store each object separately instead of flattening it as a result each object will be searched separately.
Mapping of your above mentioned document should be :
Mapping
{
"mappings": {
"properties": {
"shortDescription": {
"type": "text"
},
"audiences": {
"type": "nested"
},
"description": {
"type": "text"
},
"modules": {
"type": "text"
},
"preparation": {
"type": "text"
},
"owner": {
"properties": {
"scope": {
"type": "text"
},
"organisationalUnit": {
"type": "text"
},
"profession": {
"type": "text"
},
"supplier": {
"type": "text"
}
}
},
"learningProvider": {
"properties": {
"id": {
"type": "text"
},
"name": {
"type": "text"
},
"termsAndConditions": {
"type": "text"
},
"cancellationPolicies": {
"type": "text"
}
}
},
"visibility": {
"type": "text"
},
"status": {
"type": "text"
},
"topicId": {
"type": "text"
}
}
}
}
Now, if we index this document :
Document
{
"shortDescription": "asdf",
"description": "asdf",
"learningOutcomes": "",
"modules": [],
"learningProvider": {
"id": "ig2-zIY_QkSpMC4O0Lm0hw",
"name": null,
"termsAndConditions": [],
"cancellationPolicies": []
},
"audiences": [
{
"id": "VfDpsS_5SXi8iZubzTkUBQ",
"name": "comm",
"areasOfWork": [
"Communications"
],
"departments": [],
"grades": [
"G6"
],
"interests": [],
"requiredBy": null,
"frequency": null,
"type": "OPEN",
"eventId": null
},
{
"id": "eZPPPqTqRdiDAE3xCPlJMQ",
"name": "analysis",
"areasOfWork": [
"Analysis"
],
"departments": [],
"grades": [
"G7"
],
"interests": [],
"requiredBy": null,
"frequency": null,
"type": "REQUIRED",
"eventId": null
}
],
"preparation": "",
"owner": {
"scope": "LOCAL",
"organisationalUnit": "co",
"profession": 63,
"supplier": ""
},
"visibility": "PUBLIC",
"status": "Published",
"topicId": ""
}
If you search query is this :
Search Query 1
:
{
"query": {
"nested": {
"path": "audiences",
"query": {
"bool": {
"must": [
{
"match": {
"audiences.type.keyword": "OPEN"
}
},
{
"match": {
"audiences.grades.keyword": "G6"
}
}
]
}
}
}
}
}
Result
"hits": [
{
"_index": "product",
"_type": "_doc",
"_id": "1",
"_score": 0.9343092,
"_source": {
"shortDescription": "asdf",
"description": "asdf",
"learningOutcomes": "",
"modules": [],
"learningProvider": {
"id": "ig2-zIY_QkSpMC4O0Lm0hw",
"name": null,
"termsAndConditions": [],
"cancellationPolicies": []
},
"audiences": [
{
"id": "VfDpsS_5SXi8iZubzTkUBQ",
"name": "comm",
"areasOfWork": [
"Communications"
],
"departments": [],
"grades": [
"G6"
],
"interests": [],
"requiredBy": null,
"frequency": null,
"type": "OPEN",
"eventId": null
},
{
"id": "eZPPPqTqRdiDAE3xCPlJMQ",
"name": "analysis",
"areasOfWork": [
"Analysis"
],
"departments": [],
"grades": [
"G7"
],
"interests": [],
"requiredBy": null,
"frequency": null,
"type": "REQUIRED",
"eventId": null
}
],
"preparation": "",
"owner": {
"scope": "LOCAL",
"organisationalUnit": "co",
"profession": 63,
"supplier": ""
},
"visibility": "PUBLIC",
"status": "Published",
"topicId": ""
}
}
]
But now if your search query is :
Search Query 2 :
{
"query": {
"nested": {
"path": "audiences",
"query": {
"bool": {
"must": [
{
"match": {
"audiences.type.keyword": "OPEN"
}
},
{
"match": {
"audiences.grades.keyword": "G7"
}
}
]
}
}
}
}
}
Result :
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
So, in short you need to change datatype of audiences field in your mapping and your rest query as well so that it can search for nested datatype.
So, instead of this code fragment :
BoolQueryBuilder filterQuery = boolQuery();
filterQuery.must(QueryBuilders.matchQuery("audiences.grades.keyword", "G6"));
filterQuery.must(QueryBuilders.matchQuery("audiences.type", "OPEN"));
you should use this nested query :
BoolQueryBuilder filterQuery = new BoolQueryBuilder();
filterQuery.must(QueryBuilders.matchQuery("audiences.grades.keyword", "G6"));
filterQuery.must(QueryBuilders.matchQuery("audiences.type", "OPEN"));
NestedQueryBuilder nested = new NestedQueryBuilder("audiences", filterQuery, ScoreMode.None);
Upvotes: 2
Related Questions
- Searching Multiple Fields in ElasticSearch Query
- elasticsearch search - search multiple fields
- How do I search one field with multiple values in elasticsearch?
- Search for multiple values of same field in elasticsearch
- How to search in multiple fields in elasticsearch?
- How to query on multiple fields in elasticsearch?
- Elasticsearch multiple fields search (AND OR)
- Elastic Search: include multiple field in query
- In elasticsearch I'm trying to multi search multiple fields with multiple values is it possible?
- Searching multiple fields in elasticsearch