CODEWITHSUNDEEP
elasticsearch
Eser Aygün
Eser Aygün

Reputation: 8014

Listening to recently added documents that matches a given query

I need to listen to a subset of documents added to an Elasticsearch index. The subset is defined by an Elasticsearch query. I can tolerate a 10 second delay between the index operation and the listening callback.

Obviously, I can do this by sending search requests to the server every 10 seconds. But searching the entire index for recent documents seems redundant. I can store the id of the last document I've fetched and use that to narrow down the search further, which I will do if there is no easier way.

I thought, however, there may be a plugin that will catch any newly inserted document, try to match the query against the document and push it to my listener if the match was successful. Does such plugin exists? Is this at least possible?

Upvotes: 2

Views: 2075

Answers (2)

knutwalker
knutwalker

Reputation: 5974

You can take a look at the Percolator feature, which does what you described. Also, there are significant changes in the upcoming 1.0 release, see: Master Documentation

Edit: As of Elasticsearch 5.0, the Percolator has been deprecated and replaced with the percolator query

Upvotes: 1

Haiyuan Zhang
Haiyuan Zhang

Reputation: 42832

the delay depends on how many queries you need to match. recently, I've test the percolate feature in the 1.0 beta version. to percolate one document via 100 registered queries, less than 10ms, 1000, above 15ms, 10000 above 100ms, seems the delay increase linearly with the number of registered queries. very bad. after reading "how does it work under the hook" section of beta 1.0 percolate documentation, I'm confirmed that the "percolating" is done in a linear way.

Upvotes: 0

Related Questions