How to build an Elasticsearch-Query with startsWith-functionality and special characters

Question

I have JsonObjects that i search with Elasticsearch from a Java Application, using the Java API to build searchQueries. The objects contain a field called "such" that contains a searchString with which the JsonObject should be found, for example a simple searchString would be "STVBBM160A". Besides the usual characters a-Z 0-9 the searchString could also look like the following examples: "STV-157ABR", "F-G/42-W3" or "DDM000.074.6652"

The search should return results already when only the first characters are put into a searchfield, which it does for a search like "F-G/42"

My Problem: The search sometimes doesn't return results at all, but when the last character is typed it finds the right document.

What i tried: First I wanted to use a WildcardQuery where the query would be "typedStuff*", but the WildcardQuery didn't return any results at all, as soon as I typed anything but * (It used to work for other searchFields with other values)

Now I am using a QueryStringQuery, which also takes the input and puts a * character to the end. By escaping the QueryString, I am able to search for Strings like "F-G/42" and so on, but the search for "DDM000.074.6652" doesn't return any results until elasticsearch has the whole String to search. Also, when i type "STV" all results with "STV-xxxxx" (containing the "-" after STV) are returned, but not the object with "STVBBM160A", again until the whole String is given for the search (without showing any results inbetween as soon as the searchString is "STVB")

This is the query I'm using right now:

{
  "size": 1000,
  "min_score": 1,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "MY_DATA_TYPE",
            "fields": [
              "doc.db_doc_type"
            ]
          }
        },
        {
          "query_string": {
            "query": "MY_SPECIFIC_TYPE",
            "fields": [
              "doc.db_doc_specific"
            ]
          }
        }
      ],
      "should": {
        "query_string": {
          "query": "STV*",
          "fields": [
            "doc.such"
          ],
          "boost": 3,
          "escape": true
        }
      }
    }
  }
}

This is the old Query with the WildCardQuery, which doesn't return any results at all unless there is no queryString but *:

{
  "size": 50,
  "min_score": 1,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "MY_DATA_TYPE",
            "fields": [
              "doc.db_doc_type"
            ]
          }
        },
        {
          "query_string": {
            "query": "MY_SPECIFIC_TYPE",
            "fields": [
              "doc.db_doc_specific"
            ]
          }
        }
      ],
      "should": {
        "wildcard": {
          "doc.such": {
            "wildcard": "STV*",
            "boost": 3
          }
        }
      }
    }
  }
}

When using a PrefixQuery, the search also doesn't return any results at all (with and without the *):

{
  "size": 50,
  "min_score": 1,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "MY_DATA_TYPE",
            "fields": [
              "doc.db_doc_type"
            ]
          }
        },
        {
          "query_string": {
            "query": "MY_SPECIFIC_TYPE",
            "fields": [
              "doc.db_doc_specific"
            ]
          }
        }
      ],
      "should": {
        "prefix": {
          "doc.such": {
            "prefix": "HSTKV*",
            "boost": 3
          }
        }
      }
    }
  }
} 

How can this query be changed to achieve the goal of getting all results starting with the specified String, no matter if the field doc.such also contains Numbers or special chars like "_" or "." or "/" ?

Thanks in advance

Answer 1

As soon as you want to query prefixes, suffixes or substring in a serious way, you need to leverage nGrams. In your case, since you're only after prefixes, an edgeNGram tokenizer would be in order. You need to change the settings of your index to be like this one:

PUT your_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "prefix_analyzer": {
          "tokenizer": "prefix_tokenizer",
          "filter": [
            "lowercase"
          ]
        },
        "search_prefix_analyzer": {
          "tokenizer": "keyword",
          "filter": [
            "lowercase"
          ]
        }
      },
      "tokenizer": {
        "prefix_tokenizer": {
          "type": "edgeNGram",
          "min_gram": "1",
          "max_gram": "25"
        }
      }
    }
  },
  "mappings": {
    "your_type": {
      "properties": {
        "doc": {
          "properties": {
            "such": {
              "type": "string",
              "fields": {
                "starts_with": {
                  "type": "string",
                  "analyzer": "prefix_analyzer",
                  "search_analyzer": "search_prefix_analyzer"
                }
              }
            }
          }
        }
      }
    }
  }
}

What will happen with this analyzer is that when indexing F-G/42-W3 the following tokens will be indexed: f, f-, f-g, f-g/, f-g/4, f-g/42, f-g/42-, f-g/42-w, f-g/42-w3.

At search time, we'll simply lowercase the user input and the prefix will be matched against the indexed tokens.

Then your query can simply be transformed to a match query:

{
  "size": 1000,
  "min_score": 1,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "MY_DATA_TYPE",
            "fields": [
              "doc.db_doc_type"
            ]
          }
        },
        {
          "query_string": {
            "query": "MY_SPECIFIC_TYPE",
            "fields": [
              "doc.db_doc_specific"
            ]
          }
        }
      ],
      "should": {
        "match": {
          "doc.such": {
            "query": "F-G/4"
          }
        }
      }
    }
  }
}