CODEWITHSUNDEEP
elasticsearchloggingkibanafluenttd-agent
S Andrew
S Andrew

Reputation: 7268

Fluentd not able to upload logs to Elasticsearch

I have downloaded Elasticsearch and Kibana on my Ubuntu 18.04 machine. Both of them are running fine and I can access them. Below is the Elasticsearch details:

{
  "name" : "TX-G1-000",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AobC_iiNSyyNftYl3pUJ7w",
  "version" : {
    "number" : "7.14.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I have also installed Fluentd and its service td-agent is running fine.

● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
   Loaded: loaded (/lib/systemd/system/td-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-09-20 21:18:12 IST; 7min ago
     Docs: https://docs.treasuredata.com/display/public/PD/About+Treasure+Data%27s+Server-Side+Agent
  Process: 5486 ExecStop=/bin/kill -TERM ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 5491 ExecStart=/opt/td-agent/bin/fluentd --log $TD_AGENT_LOG_FILE --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 5498 (fluentd)
    Tasks: 12 (limit: 4915)
   CGroup: /system.slice/td-agent.service
       ├─5498 /opt/td-agent/bin/ruby /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
       └─5501 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid --under-supervisor

Sep 20 21:18:11 TX-G1-000 systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Sep 20 21:18:12 TX-G1-000 systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.

Below is my td-agent.conf file:

<source>
  @type tail
  path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
  pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos
  format json
  time_format %Y-%m-%d %H:%M:%S
  tag health01
</source>

<source>
  @type tail
  path /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
  pos_file /home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos
  format json
  time_format %Y-%m-%d %H:%M:%S
  tag cycle01
</source>

<match health*>
  @type elasticsearch
  hosts http://localhost:9200/
  index_name health_skl_device
  type_name health
</match>

<match cycle*>
  @type elasticsearch
  hosts http://localhost:9200/
  index_name cycle_skl_device
  type_name cycle
</match>

When running the td-agent, below are its logs:

2021-09-20 21:18:12 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-09-20 21:18:12 +0530 [info]: gem 'fluentd' version '1.13.3'
2021-09-20 21:18:12 +0530 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json"
    pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json.pos"
    format json
    time_format %Y-%m-%d %H:%M:%S
    tag "health01"
    <parse>
      time_format %Y-%m-%d %H:%M:%S
      @type json
      unmatched_lines 
      time_type string
    </parse>
  </source>
  <source>
    @type tail
    path "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json"
    pos_file "/home/user/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json.pos"
    format json
    time_format %Y-%m-%d %H:%M:%S
    tag "cycle01"
    <parse>
      time_format %Y-%m-%d %H:%M:%S
      @type json
      unmatched_lines 
      time_type string
    </parse>
  </source>
  <match health*>
    @type elasticsearch
    hosts "http://localhost:9200/"
    index_name "health_skl_device"
    type_name "health"
  </match>
  <match cycle*>
    @type elasticsearch
    hosts "http://localhost:9200/"
    index_name "cycle_skl_device"
    type_name "cycle"
  </match>
</ROOT>
2021-09-20 21:18:12 +0530 [info]: starting fluentd-1.13.3 pid=5491 ruby="2.7.4"
2021-09-20 21:18:12 +0530 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2021-09-20 21:18:13 +0530 [info]: adding match pattern="health*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding match pattern="cycle*" type="elasticsearch"
2021-09-20 21:18:13 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: adding source type="tail"
2021-09-20 21:18:13 +0530 [info]: #0 starting fluentd worker pid=5501 ppid=5498 worker=0
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_cycle.json
2021-09-20 21:18:13 +0530 [info]: #0 following tail of /home/thingtrax/PycharmProjects/Td-Agent/logs/TX-S2-SKL-001_health.json
2021-09-20 21:18:13 +0530 [info]: #0 fluentd worker is now running worker=0

I do not see any error logs but I am not sure why it's not able to upload data. I try to create index pattern, it doesn't match on Kibana. How can I debug this issue?

Logs after adding debug

2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name health_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: health_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'index_name cycle_skl_device' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: cycle_skl_device' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-09-23 07:41:50 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: Need substitution: false
2021-09-23 07:41:50 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2021-09-23 07:41:50 +0530 [debug]: No fluent logger for internal event

Upvotes: 0

Views: 2921

Answers (1)

yurish
yurish

Reputation: 919

I think you have incorrect match tags. Nowhere in documentation does it mention that asterisks can be used that way, they should either take a place of a whole tag part or be used inside a regular expression. According to this section, Fluentd accepts all non-period characters as a part of a tag. So in fact health* is a valid name for a tag, fluentd expects exact matches of that string.

You should try using /health.*/ and /cycle.*/ instead.

Better yet, you can go the intended way, change the tag names to health.01 and cycle.01 and use health.** and cycle.** for matching.

Upvotes: 0

Related Questions