CODEWITHSUNDEEP
pythonelasticsearchelasticsearch-aggregation
moritz
moritz

Reputation: 314

Count all occurrences per field in one index

I want to count how many entries i have for each field in my elasticsearch DB for one index. I have tried with the code below, but this only returns the total number of entries. I'm working in Python.

What I have tried so far:

qry = {
"aggs": {
"field": {
"terms" : {"field": "field"}
}
}, "size": 0
}

r = es.search(body=qry,
              index="webhose_english")

My current result:

Out[64]: 
{'_shards': {'failed': 0, 'skipped': 0, 'successful': 5, 'total': 5},
'aggregations': {'field': {'buckets': [],
'doc_count_error_upper_bound': 0,
'sum_other_doc_count': 0}},
'hits': {'hits': [], 'max_score': 0.0, 'total': 4519134},
'timed_out': False,
'took': 16}

And I would ideally have something like:

{'field_1': 321,
 'field_2': 231,
 'field_3': 132}

Upvotes: 3

Views: 1434

Answers (2)

moritz
moritz

Reputation: 314

I did it by iterating over the following query and then collecting the "total" values in a dictionary:

qry = {
"query": {
"exists": {
"field": "fields_to_iterate"
}
}
}

Upvotes: 0

xeraa
xeraa

Reputation: 10859

This information used to be part of the _field_stats API, but it has been removed in 6.0. So you are on the right track, you will need an aggregation. I think value_count is the one you need and for good measure I've added global as well, so we know how many documents are there in total.

Three sample docs:

PUT foo/_doc/1
{
  "foo": "bar"
}
PUT foo/_doc/2
{
  "foo": "bar",
  "bar": "bar"
}
PUT foo/_doc/3
{
  "foo": "bar",
  "bar": "bar",
  "baz": "bar"
}

Aggregation (I'm not sure if there might be a shorter version of this especially with many fields):

GET foo/_search
{
  "aggs": {
    "count_fields": {
      "global": {},
      "aggs": {
        "count_foo": {
          "value_count": {
            "field": "foo.keyword"
          }
        },
        "count_bar": {
          "value_count": {
            "field": "bar.keyword"
          }
        },
        "count_baz": {
          "value_count": {
            "field": "baz.keyword"
          }
        }
      }
    }
  },
  "size": 0
}

Result:

{
  "took" : 16,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "count_fields" : {
      "doc_count" : 3,
      "count_foo" : {
        "value" : 3
      },
      "count_bar" : {
        "value" : 2
      },
      "count_baz" : {
        "value" : 1
      }
    }
  }
}

Upvotes: 3

Related Questions