Get the most frequent terms of text field

Question

How do I get a list of all individual tokens of a text field along with their document frequency. I want this to build a domain specific list of frequent (and therefore useless) stop words.

This question covers all the methods I found so far but

• "keyword" data type is not an option because im interested in individual terms (so tokenisation is necessary)
• "significant term aggregation" is not an option because im interested in the most frequent, not the most significant terms
• "termvector" is not an option because I need it for the hole index not just a particular document or a small subset.

Answer 1

You will have to enable field_data on your field to do this. But be careful it can impact a lot the heap memory used.

https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html