CODEWITHSUNDEEP
elasticsearchkubernetes
Sathish Kumar
Sathish Kumar

Reputation: 2556

ElasticSearch 8 errors with Action/metadata line [1] contains an unknown parameter [_type] status:400

I am trying to setup EFK (ElasticSearch 8, FluentD and Kibana) stack on K8S cluster (on-premises)

I followed this link to install elasticsearch and installed it using helm charts and followed this link to install fluentd

Output of fluentd and elasticsearch pods

[root@ctrl01 ~]#  kubectl get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                                   1/1     Running   0          136m

[root@ctrl01 ~]#  kubectl get pods -n kube-system
NAME                                                            READY   STATUS    RESTARTS   AGE
fluentd-cnb7p                                                   1/1     Running   0          107m
fluentd-dbxjk                                                   1/1     Running   0          107m

However, elasticsearch log was piled up with the following warning messages

2021-10-18 12:13:12 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2021-10-18 12:13:42 +0000 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" plugin_id="out_es"
2021-10-18 12:13:12 +0000 [warn]: suppressed same stacktrace

Conf file (tailored output)

2021-10-18 12:09:10 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type tail
    @id in_tail_container_logs
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    tag kubernetes.*
    read_from_head true
    format json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </source>
  <source>
    @type tail
    @id in_tail_minion
    path /var/log/salt/minion
    pos_file /var/log/fluentd-salt.pos
    tag salt
    format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S
  </source>

I am not sure which 'type' field it refers to. I am unable to find an example of ElasticSearch 8 for match and source directives to compare

It seems type field is not supported from ES 8 onwards but I am not sure on that. Kindly let me know the reason for the error

Upvotes: 9

Views: 24918

Answers (3)

M&#199;T
M&#199;T

Reputation: 1143

FluentD (and Fluent Bit) Elasticsearch output plugin puts a _type field in logs when transferring them to Elasticsearch. However, this field was a special field of Elasticsearch and it was removed from Elasticsearch in version 8.

Therefore, if your Elasticsearch installation has a version of 8 or later, you need to suppress FluentD (or Fluent Bit) to not put the _type field in logs by setting Suppress_Type_Name configuration parameter as On (default is Off).

[OUTPUT]
    Name  es
    Host  192.168.2.3
    Port  9200
    ...
    ... (other es output plugin parameters)
    ...
    Suppress_Type_Name On

By using this configuration parameter, FluentD (or Fluent Bit) will not include a _type field in the final log and Elasticsearch will not complain.

Reference:

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#action-metadata-contains-an-unknown-parameter-type

Upvotes: 5

Aman_Bhala
Aman_Bhala

Reputation: 317

I faced similar errors when I tried to use elasticsearch 8.2.3 with fluentBit 1.9.5. I could see elastic was sending logs but could not see any data in kibana webpage due to which could not create indices and saw the above error in fluent-bit pod logs. I followed this github issue and added Suppress_Type_Name On under outputs: section in my fluent-bit helm chart values.yaml file and it worked fine after that.

      [OUTPUT]
          Name  es
          Match *
          Host  {{ .Values.global.backend.es.host }}
          Port  {{ .Values.global.backend.es.port }}
          Logstash_Format Off
          Retry_Limit False
          Type  _doc
          Time_Key @timestamp
          Replace_Dots On
          Suppress_Type_Name On
          Index {{ .Values.global.backend.es.index }}
      {{ .Values.extraEntries.output }}

Upvotes: 15

dogukanarkan
dogukanarkan

Reputation: 351

I was working on the same issue for a few days and I found a solution but just a workaround, not the optimal solution.

If you set TypeName as null for ElasticsearchSinkOptions, you don't face this issue.

Unfortunately, you can't set it from appsettings.json. At least I couldn't find a way.

In background, Serilog.Sinks.ElasticSearch library use this property as _type in HTTP header. But the '_type' header, as leandrojmp pointed out in the comment, it is no longer available in version 8.2 of ElasticSearch.

Upvotes: 5

Related Questions