CODEWITHSUNDEEP
elasticsearch
igx
igx

Reputation: 4231

How to get all field names in elasticsearch index

I just start using elasticsearch 5.2 .

I am trying to get all keys in the index if I have the following mapping:

"properties": {
         "name": { "type": "text" },
         "article": {
          "properties": {
           "id": { "type": "text" },
           "title":  { "type": "text"},
           "abstract": { "type": "text"},
            "author": {
             "properties": {
              "id": { "type": "text" },
              "name": { "type": "text" }
}}}} } }

is it possible to get all fields full name ? like this:

 name,
 article.id ,
 article.title ,
 article.abstract ,
 article.author.id,
 article.author.name

how can I get that ?

Upvotes: 7

Views: 20757

Answers (3)

Droid
Droid

Reputation: 1362

You can try this, Get Field Mapping API

def unique_preserving_order(sequence):
    """
    Preserving Order
    :param sequence: object list
    :return:  new list from the set’s contents
    """

    seen = set()
    return [x for x in sequence if not (x in seen or seen.add(x))]

get es index fields recursively 

def get_fields_recursively(dct, field_types=None):

    if dct and 'properties' in dct:
        fields = []
        for key, ndct in dct.get('properties').items():
            if 'properties' in ndct:
                for nkey, nd in ndct.items():
                    if nkey == 'properties':
                        field = get_fields_recursively(ndct)

                        if field_types:
                            for f in field:
                                prop = ndct.get('properties').get(f)
                                if prop and prop.get('type') in field_types:
                                    ff = '{0}.{1}'.format(key, f)
                                    # fields.append(f)
                                    fields.append(ff)
                        else:
                            # if not key.startswith('@'):
                            # _fields = field + ['{0}.{1}'.format(key, f) for f in field]
                            _fields = ['{0}.{1}'.format(key, f) for f in field]
                            fields.extend(_fields)
                        continue

                continue

            if field_types:
                if ndct.get('type') in field_types and not key.startswith('@'):
                    fields.append(key)
            else:
                if not key.startswith('@'):
                    fields.append(key)
        return fields
    else:
        return dct

get fields from index mappings, also you can filter fields by types, ex. text fields or numerical fields

def get_mapping_fields(self, field_type=None, index=None, params={}):
    """

    :param field_type: es field types, filter fields by type
    :param index: elastic index name
    :param params: mapping additional params
    :return: fields

    <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-field-mapping.html>
    - http://eshost:9200/_mapping
    - http://eshost:9200/_all/_mapping
    - http://eshost:9200/index_name/_mapping

    """

    _fields = []
    _mapping = self.esclient.indices.get_mapping(index=index, params=params)
    for idx_mapping in _mapping.values():
        mapping = idx_mapping.get('mappings')
        if 'system' in mapping:
            mapping = mapping.get('system')
        else:
            mapping = mapping.get('doc')
        fields = get_fields_recursively(mapping, field_type)
        if fields:
            _fields.extend(fields)

    return list(unique_preserving_order(_fields))

Upvotes: 0

redlus
redlus

Reputation: 2491

The mapping API also allows querying the field names directly. Here is a python 3 code snippet which should do the work:

import json
import requests

# get mapping fields for a specific index:
index = "INDEX_NAME"
elastic_url = "http://ES_HOSTNAME:9200"
doc_type = "DOC_TYPE"
mapping_fields_request = "_mapping/field/*?ignore_unavailable=false&allow_no_indices=false&include_defaults=true"
mapping_fields_url = "/".join([elastic_url, index, doc_type, mapping_fields_request])
response = requests.get(mapping_fields_url)

# parse the data:
data = response.content.decode()
parsed_data = json.loads(data)
keys = sorted(parsed_data[index]["mappings"][doc_type].keys())
print("index= {} has a total of {} keys".format(index, len(keys)))

# print the keys of the fields:
for i, key in enumerate(keys):
    if i % 43 == 0:
        input()
    print("{:4d}:     {}".format(i, key))

Very convenient indeed. Do note that keys which contain "." in their name may confuse you a bit in how cascaded they are in the document...

Upvotes: 3

Rahul
Rahul

Reputation: 16335

You may use _field_names field.

The _field_names field indexes the names of every field in a document that contains any value other than null.

GET _search
{
  "size"0,
  "aggs": {
    "Field names": {
      "terms": {
        "field": "_field_names", 
        "size": 100
      }
    }
  }
}

Update : from ES 5 onwards

the _field_names field has been locked down and is only indexed, it doesn't support fielddata (memory intensive) or doc values,

Ref : https://github.com/elastic/elasticsearch/issues/22576

As an alternative, you may getMapping API

The get mapping API can be used to get more than one index or type mapping with a single call. General usage of the API follows the following syntax: host:port/{index}/_mapping/{type}

$ curl -XGET 'http://localhost:9200/index/_mapping?pretty'

You may then process the response to extract all the field names in the index

Upvotes: 3

Related Questions