How to find entity in search query in Elasticsearch?

Question

I'm using Elasticsearch to build search for ecommerece site.

One index will have products stored in it, in products index I'll store categories in it's other attributes along with. Categories can be multiple but the attribute will have single field value. (E.g. color)

Let's say user types in Black(color) Nike(brand) shoes(Categories)

I want to process this query so that I can extract entities (brand, attribute, etc...) and I can write Request body search.

I have tought of following option,

1. Applying regex on query first to extract those entities (But with this approach not sure how Fuzzyness would work, user may have typo in any of the entity)

2. Using OpenNLP extension (But this one only works on indexation time, in above scenario we want it on query side)

3. Using NER of any good NLP framework. (This is not time & cost effective because I'll have millions of products in engine also they get updated/added on frequent basis)

What's the best way to solve above issue ?

---

Edit:

1. Found couple of libraries which would allow fuzzy text matching in regex. But the entities to find will be many, so what's the best solution to optimise that ?

2. Still not sure about OpenNLP

3. NER won't work in this case because there are fixed number of entities so prediction is not right when there are no entity available in the query.

Answer 1

I would approach this problem as NER tagging considering you already have corpus of tags. My approach for this problem will be as below:

1. Create a annotated dataset of queries with each word tagged to one of the tags say {color, brand, Categories}
2. Train a NER model (CRF/LSTMS).

This is not time & cost effective because I'll have millions of products in engine also they get updated/added on frequent basis

To handle this situation I suggest dont use words in the query as features but rather use the attributes of the words as features. For example create an indicator function f(x',y) for word x with context x' (i.e the word along with the surrounding words and their attributes) and tag y which will return a 1 or 0. A sample indicator function will be as below

f('blue', 'y') = if 'blue' in `color attribute` column of DB and words previous to 'blue' is in `product attribute` column of DB and 'y' is `colors` then return 1 else 0.

Create lot of these indicator functions also know as features maps.

These indicator functions are then used to train a models using CRFS or LSTMS. Finially we use viterbi algorithm to find the best tagging sequence for your query. For CRFs you can use packages like CRFSuite or CRF++. Using these packages all you have go do is create indicator functions and the package will train a model for you. Once trained you can use this model to predict the best sequence for your queries. CRFs are very fast.

This way of training without using vector representation of words will generalise your model without the need of retraining. [Look at NER using CRFs].

Answer 2

If you cannot achieve desired results with tuning of built-in ElasticSearch scoring/boosting most likely you'll need some kind of 'natural language query' processing:

1. Tokenize free-form query. Regex can be used for splitting lexems, however very often it is better to write custom tokenizer for that.
2. Perform named-entity recognition to determine possible field(s) for each keyword. At this step you will get associations like (Black -> color), (Black -> product name) etc. In fact you don't need OpenNLP for that as this should be just an index (keyword -> field(s)), and you can try to use ElasticSearch 'suggest' API for this purpose.
3. (optional) Recognize special phrases or combinations like "released yesterday", "price below $20"
4. Generate possible combinations of matches, and with help of special scoring function determine 'best' recognition result. Scoring function may be hardcoded (reflect 'common sense' heuristics) or it this may be a result of machine learning algorithm.
5. By recognition result (matches metadata) produce formal query to produce search results - this may be ElasticSearch query with field hints, or even SQL query.

In general, efficient NLQ processing needs significant development efforts - I don't recommend to implement it from scratch until you have enough resources & time for this feature. As alternative, you can try to find existing NLQ solution and integrate it, but most likely this will be commercial product (I don't know any good free/open-source NLQ components that really ready for production use).