CODEWITHSUNDEEP
elasticsearchkibana
Gurpreet Singh
Gurpreet Singh

Reputation: 21233

How to apply custom analyser?

Just discovered an issue with our Elastic Search. It is not returning anything for '&' in field name. Did some googling and I think I need a custom analyser. Never worked with ES before, assumption is I'm missing something basic here.

This is what I have got and it is not working as expected. 

PUT custom_analyser
{
 "settings": {
    "analysis": {
      "analyzer": {
        "suggest_analyzer": {
          "type":      "custom",
          "tokenizer": "whitespace",
          "filter":    [ "lowercase", "my_synonym_filter" ]
        }
      },
      "filter": {
        "my_synonym_filter": {
          "type": "synonym", 
          "synonyms": [
              "&, and",
              "foo, bar" ]
        }
      }
    }
  }
}

And trying to use it like: 

GET custom_analyser/_search
{
  "aggs": {
    "section": {
      "terms": {
        "field": "section",
        "size": 10,
        "shard_size": 500,
        "include": "jill & jerry" //Not returning anything back for this field using default analyser
      }
    }
  }
}

Output: 

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "section": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": []
    }
  }
}

Mappings

 "_doc":{  
      "dynamic":"false",
      "date_detection":false,
      "properties":{  
         "section":{  
            "type":"keyword"
         }
      }
   }

GET custom_analyser:

{
  "custom_analyser": {
    "aliases": {},
    "mappings": {},
    "settings": {
      "index": {
        "number_of_shards": "5",
        "provided_name": "custom_analyser",
        "creation_date": "1565971369814",
        "analysis": {
          "filter": {
            "my_synonym_filter": {
              "type": "synonym",
              "synonyms": [
                "&, and",
                "foo, bar"
              ]
            }
          },
          "analyzer": {
            "suggest_analyzer": {
              "filter": [
                "lowercase",
                "my_synonym_filter"
              ],
              "type": "custom",
              "tokenizer": "whitespace"
            }
          }
        },
        "number_of_replicas": "1",
        "uuid": "oVMOU5wPQ--vKhE3dDFG2Q",
        "version": {
          "created": "6030199"
        }
      }
    }
  }
}

Upvotes: 0

Views: 62

Answers (1)

xeraa
xeraa

Reputation: 10859

I think there is a slight confusion here: An analyzer won't help you, because you are (correctly) using a keyword field for the aggregation, but those are not analyzed. You could only use a normalizer on those fields.

For your specific problem: The include (and exclude) are regular expressions — you'll need to escape the & to make this work as expected.

Full example

Mapping and sample data:

PUT test
{
  "mappings": {
    "properties": {
      "section": {
        "type": "keyword"
      }
    }
  }
}

PUT test/_doc/1
{
  "section": "jill & jerry"
}
PUT test/_doc/2
{
  "section": "jill jerry"
}
PUT test/_doc/3
{
  "section": "jill"
}
PUT test/_doc/4
{
  "section": "jill & jerry"
}

Query — you need a double backslash for the escape to work here (and I'm also excluding the actual documents with "size": 0 to keep the response shorter):

GET test/_search
{
  "size": 0,
  "aggs": {
    "section": {
      "terms": {
        "field": "section",
        "include": "jill \\& jerry"
      }
    }
  }
}

Response:

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "section" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "jill & jerry",
          "doc_count" : 2
        }
      ]
    }
  }
}

Upvotes: 1

Related Questions