CODEWITHSUNDEEP
searchelasticsearch
kiran
kiran

Reputation: 497

Query all unique values of a field with Elasticsearch

How do I search for all unique values of a given field with Elasticsearch?

I have such a kind of query like select full_name from authors, so I can display the list to the users on a form.

Upvotes: 39

Views: 52064

Answers (5)

Dhivya Dandapani
Dhivya Dandapani

Reputation: 431

Intuition: In SQL parlance:

Select distinct full_name from authors;

is equivalent to

Select full_name from authors group by full_name;

So, we can use the grouping/aggregate syntax in ElasticSearch to find distinct entries.

Assume the following is the structure stored in elastic search :

[{
    "author": "Brian Kernighan"
  },
  {
    "author": "Charles Dickens"
  }]

What did not work: Plain aggregation

{
  "aggs": {
    "full_name": {
      "terms": {
        "field": "author"
      }
    }
  }
}

I got the following error: 

{
  "error": {
    "root_cause": [
      {
        "reason": "Fielddata is disabled on text fields by default...",
        "type": "illegal_argument_exception"
      }
    ]
  }
}

What worked like a charm: Appending .keyword with the field

{
  "aggs": {
    "full_name": {
      "terms": {
        "field": "author.keyword"
      }
    }
  }
}

And the sample output could be:

{
  "aggregations": {
    "full_name": {
      "buckets": [
        {
          "doc_count": 372,
          "key": "Charles Dickens"
        },
        {
          "doc_count": 283,
          "key": "Brian Kernighan"
        }
      ],
      "doc_count": 1000
    }
  }
}

Bonus tip:

Let us assume the field in question is nested as follows:

[{
    "authors": [{
        "details": [{
            "name": "Brian Kernighan"
          }]
      }]
  },
  {
    "authors": [{
        "details": [{
            "name": "Charles Dickens"
          }]
      }]
  }
]

Now the correct query becomes:

{
  "aggregations": {
    "full_name": {
      "aggregations": {
        "author_details": {
          "terms": {
            "field": "authors.details.name"
          }
        }
      },
      "nested": {
        "path": "authors.details"
      }
    }
  },
  "size": 0
}

Upvotes: 6

kgf3JfUtW
kgf3JfUtW

Reputation: 14926

Working for Elasticsearch 5.2.2

curl -XGET  http://localhost:9200/articles/_search?pretty -d '
{
    "aggs" : {
        "whatever" : {
            "terms" : { "field" : "yourfield", "size":10000 }
        }
    },
    "size" : 0
}'

The "size":10000 means get (at most) 10000 unique values. Without this, if you have more than 10 unique values, only 10 values are returned.

The "size":0 means that in result, "hits" will contain no documents. By default, 10 documents are returned, which we don't need.

Reference: bucket terms aggregation

Also note, according to this page, facets have been replaced by aggregations in Elasticsearch 1.0, which are a superset of facets.

Upvotes: 6

Serp C
Serp C

Reputation: 883

The existing answers did not work for me in Elasticsearch 5.X, for the following reasons:

  • I needed to tokenize my input while indexing.
  • "size": 0 failed to parse because "[size] must be greater than 0."
  • "Fielddata is disabled on text fields by default." This means by default you cannot search on the full_name field. However, an unanalyzed keyword field can be used for aggregations.

Solution 1: use the Scroll API. It works by keeping a search context and making multiple requests, each time returning subsequent batches of results. If you are using Python, the elasticsearch module has the scan() helper function to handle scrolling for you and return all results.

Solution 2: use the Search After API. It is similar to Scroll, but provides a live cursor instead of keeping a search context. Thus it is more efficient for real-time requests.

Upvotes: 4

Gary Gauh
Gary Gauh

Reputation: 5135

For Elasticsearch 1.0 and later, you can leverage terms aggregation to do this,

query DSL:

{
  "aggs": {
    "NAME": {
      "terms": {
        "field": "",
        "size": 10
      }
    }
  }
}

A real example:

{
  "aggs": {
    "full_name": {
      "terms": {
        "field": "authors",
        "size": 0
      }
    }
  }
}

Then you can get all unique values of authors field. size=0 means not limit the number of terms(this requires es to be 1.1.0 or later).

Response:

{
    ...

    "aggregations" : {
        "full_name" : {
            "buckets" : [
                {
                    "key" : "Ken",
                    "doc_count" : 10
                },
                {
                    "key" : "Jim Gray",
                    "doc_count" : 10
                },
            ]
        }
    }
}

see Elasticsearch terms aggregations.

Upvotes: 17

javanna
javanna

Reputation: 60245

You could make a terms facet on your 'full_name' field. But in order to do that properly you need to make sure you're not tokenizing it while indexing, otherwise every entry in the facet will be a different term that is part of the field content. You most likely need to configure it as 'not_analyzed' in your mapping. If you are also searching on it and you still want to tokenize it you can just index it in two different ways using multi field.

You also need to take into account that depending on the number of unique terms that are part of the full_name field, this operation can be expensive and require quite some memory.

Upvotes: 19

Related Questions