CODEWITHSUNDEEP
logstash
Letokteren
Letokteren

Reputation: 809

How to use environmental variables in Logstash 5.X?

We've just switched from 2.X to 5.X, and I'm trying to find out how to use environmental variables in the pipeline configuration files.

In 2.X, the following worked:

export HOSTNAME

Then start Logstash with the --allow-env command line flag. The pipeline configuration file looked like this:

filter {
       mutate {
              add_field => { "some_field" => "${HOSTNAME}"}
       }
}

The documentation says that the --allow-env flag is not required anymore.

I've tried to replace mutate with the environment filter, but with no luck.

I've tried to edit the startup.options file. Added HOSTNAME as a usual environment variable, and added it between the read-EOM part, but without any positive results.

What seems to be working now if I add the following part to the /usr/share/logstash/bin/logstash.lib.sh file, but I'm nut sure if I'm supposed to edit that.

HOSTNAME="the-name-of-the-host"
export HOSTNAME

So my question is: what have I overlooked? What is the proper way to allow the usage of enviromental variables in Logstash's pipeline configuration files?

Note:

Like Alcanzar described, this works if I run Logstash manually. However we would like to run it with systemctl, so it should be automatically started by the daemon on startup. In 2.X, it worked fine with the /etc/init.d/logstash file, but as the documentation describes, that is no more. The files I'm supposed to edit are: /etc/logstash/startup.options and /etc/logstash/logstash.yml.

Upvotes: 4

Views: 2870

Answers (2)

Nick Ginanto
Nick Ginanto

Reputation: 32190

we had the same issue with logstash 5. The best way (that we found) is to add the env vars in /etc/systemd/system/logstash.service.d/override.conf and in there have

   [Service]
   Environment="HOSTNAME=the-name-of-the-host"

Upvotes: 1

Alcanzar
Alcanzar

Reputation: 17165

The environment variables clearly work if starting manually:

With a test.conf of this:

input {
    stdin { codec => "json" }
}
filter {
    mutate {
        add_field => {
            "hostname" => "${HOSTNAME}"
        }
    }
}

output {
    stdout { codec => "rubydebug" }
}

We can run a test and verify it:

% export HOSTNAME="abc1234" 
% echo '{"a":1}' | bin/logstash -f test.conf
Sending Logstash's logs to /Users/xxxxxx/logstash-5.1.1/logs which is now configured via log4j2.properties
[2017-02-16T09:04:33,442][INFO ][logstash.inputs.stdin    ] Automatically switching from json to json_lines codec {:plugin=>"stdin"}
[2017-02-16T09:04:33,454][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-02-16T09:04:33,457][INFO ][logstash.pipeline        ] Pipeline main started
[2017-02-16T09:04:33,492][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
             "a" => 1,
      "hostname" => "abc1234",
    "@timestamp" => 2017-02-16T15:04:33.497Z,
      "@version" => "1",
          "host" => "XXXXXX.local",
          "tags" => []
}
[2017-02-16T09:04:36,472][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}

So the real question is why isn't it working in your scenario. If you are using some sort of /etc/init.d script to start logstash, then you can add to /etc/sysconfig/logstash a line like export HOSTNAME="whatever"

Upvotes: 2

Related Questions