CODEWITHSUNDEEP
dockerdocker-composesplunk
Shaun Scovil
Shaun Scovil

Reputation: 3987

How to initialize Splunk HTTP Event Collector via Docker Compose and use it with splunk logging driver

I'm trying to set up a local development environment with Docker Compose that bootstraps a Splunk Enterprise server and uses the splunk logging driver on an app server.

Versions:

My docker-compose.yml file looks like this:

version: "3.7"

services:
  app:
    build: ./app
    command: bash -c "npm run start:docker"
    depends_on:
      - splunk
    environment:
      - NODE_ENV=development
      - SERVER_PORT=8080
    logging:
      driver: splunk
      options:
        splunk-format: "json"
        splunk-insecureskipverify: "true"
        splunk-source: "app"
        splunk-token: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
        splunk-url: "http://splunk:8088"
        tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
    ports:
      - "80:8080"
    volumes:
      - "./app:/usr/src/app"

  splunk:
    environment:
      - SPLUNK_ENABLE_LISTEN=9997
      - SPLUNK_START_ARGS=--accept-license --no-prompt --answer-yes
      - SPLUNK_USERNAME=admin
      - SPLUNK_PASSWORD=password
    hostname: splunk
    image: splunk/splunk:7.2.0
    ports:
      - "8000:8000"
      - "8088:8088"
      - "9997:9997"
    restart: always

In order for this to work as intended, I need to generate an HTTP Event Collector token and make it available to the app service somehow.

I've seen that you can use the environment variable SPLUNK_CMD to run commands, presumably after the Splunk service is up and running, but when I tried using that to generate a token with the CLI, nothing happened. I saw no failure in the logs, and no token under Settings > Data Inputs.

Another issue is that Splunk takes some time to start up, and before it starts listening the app service fails to build because the logging driver cannot connect.

Is it possible to do what I'm trying to do? If so, how?

Upvotes: 4

Views: 2729

Answers (1)

outcoldman
outcoldman

Reputation: 11832

The configuration of the new image (7.2.0) says that you can specify an HTTP Event Collector token with the environment variable https://github.com/splunk/docker-splunk/blob/48d5322bc574792a5bfbfe8f68769aa16e7688b7/documentation/ADVANCED.md#valid-enterprise-environment-variables

But I don't think it works for single instance after looking at https://github.com/splunk/splunk-ansible/search?q=set_as_hec_receiver.yml&unscoped_q=set_as_hec_receiver.yml - seems like that playbook will be executed only for heavy-weight-forwarder and indexer.

Alternatively, if you will look at the "legacy"/community supported image you will find a different way of doing that. As an example, you can take a look at the app-boilerplate that we use at Outcold Solutions for developing Splunk apps https://github.com/outcoldsolutions/splunk-app-boilerplate, where we:

  1. Map configurations https://github.com/outcoldsolutions/splunk-app-boilerplate/blob/master/Makefile#L23
  2. Copy it over https://github.com/outcoldsolutions/splunk-app-boilerplate/blob/master/Makefile#L26

To solve this issue "Another issue is that Splunk takes some time to start up, and before it starts listening the app service fails to build because the logging driver cannot connect." - please take a look on option splunk-verify-connection (see https://docs.docker.com/config/containers/logging/splunk/#splunk-options), in that way it will keep retrying to send the data over and over till the HTTP Event Collector will be available.

As alternative to splunk-verify-connection you can also use a different approach of forwarding logs to Splunk, by using Outcold Solutions collector, that forwards container logs from JSON logs. It is easy to install https://www.outcoldsolutions.com/docs/monitoring-docker/v5/installation/, and you will be able to use an application for monitoring your docker environments as well https://splunkbase.splunk.com/app/3723/

Upvotes: 2

Related Questions