Logstash combine data stream and index templates

Question

I'm trying to use data streams and index templates in logstash v7.17

What is the right elasticsearch output configuration to achieve this?

Option 1 Using data_stream in the tempalte -> FAILS

    output {
      elasticsearch {
        hosts => ["https://elasticsearch-master:9200"]
        index => "microservice-%{+YYYY.MM.dd}"
        template => "/usr/share/logstash/templates/microservices.json"
        # template_overwrite => false
        template_name => "microservices"
      }    
    }

Content of /usr/share/logstash/templates/microservices.json:

    {
      "index_patterns": "microservice-*",
      "template": {
        "settings" : {
            "index" : {
              "number_of_shards" : "1",
              "number_of_replicas" : "1"
            }
          },
        "mappings" : {
            "properties" : {
              "@timestamp" : {
                "type" : "date"
              },
              "@version" : {
                "type" : "keyword"
              },
              "host" : {
                "type" : "keyword"
              },
              "level" : {
                "type" : "keyword"
              },
              "service" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              }
            }
          }
      },
      "data_stream" : {
          "hidden" : false
      }
    }

Logstash logs (debug mode):

18:37:34.459 [[.monitoring-logstash]-pipeline-manager] INFO  logstash.outputs.elasticsearchmonitoring - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
18:37:34.470 [Ruby-0-Thread-14: :1] INFO  logstash.outputs.elasticsearchmonitoring - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
18:37:34.564 [[.monitoring-logstash]-pipeline-manager] WARN  logstash.javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
18:37:34.666 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch-master:9200/"}
18:37:34.678 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch version determined (7.16.3) {:es_version=>7}
18:37:34.737 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
18:37:34.850 [[main]-pipeline-manager] DEBUG logstash.outputs.elasticsearch - Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline. 
18:37:34.850 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
18:37:35.059 [Ruby-0-Thread-17: :1] INFO  logstash.outputs.elasticsearch - Using mapping template from {:path=>"/usr/share/logstash/templates/microservices.json"}
18:37:35.067 [Ruby-0-Thread-17: :1] DEBUG logstash.outputs.elasticsearch - Attempting to install template {:template=>{"index_patterns"=>"microservice-*", "template"=>{"settings"=>{"index"=>{"number_of_shards"=>"1", "number_of_replicas"=>"1", "refresh_interval"=>"5s"}}, "mappings"=>{"properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "host"=>{"type"=>"text", "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}, "level"=>{"type"=>"keyword"}, "service"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}}, "data_stream"=>{"hidden"=>false}}}
18:37:35.135 [[main]-pipeline-manager] INFO  logstash.javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf", "/usr/share/logstash/pipeline/uptime.conf"], :thread=>"#<Thread:0x38ca1ed0@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}
18:37:35.136 [[.monitoring-logstash]-pipeline-manager] INFO  logstash.javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x3d81d305 run>"}
18:37:35.346 [Ruby-0-Thread-17: :1] INFO  logstash.outputs.elasticsearch - Installing Elasticsearch template {:name=>"microservices"}
18:37:35.677 [Ruby-0-Thread-17: :1] ERROR logstash.outputs.elasticsearch - Failed to install template {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://elasticsearch-master:9200/_template/microservices'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:408:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:149:in `block in after_successful_connection'"]}

---

Option 2 Using data_streams in the output -> FAILS

    output {
      elasticsearch {
        hosts => ["https://elasticsearch-master:9200"]
        data_stream => true
        #data_stream_type => "logs"
        #data_stream_dataset => "microservices"
        #data_stream_namespace => ""    
        index => "microservice-%{+YYYY.MM.dd}"
        template => "/usr/share/logstash/templates/microservices.json"
        # template_overwrite => false
        # ecs_compatibility => "v1"
        template_name => "microservices"
      }    
    }

The logs are:

18:41:47.356 [[main]-pipeline-manager] ERROR logstash.outputs.elasticsearch - Invalid data stream configuration, following parameters are not supported: {"template"=>"/usr/share/logstash/templates/microservices.json", "template_name"=>"microservices", "index"=>"microservice-%{+YYYY.MM.dd}"}
18:41:47.357 [Ruby-0-Thread-16: :1] ERROR logstash.outputs.elasticsearch - Invalid data stream configuration, following parameters are not supported: {"template"=>"/usr/share/logstash/templates/microservices.json", "template_name"=>"microservices", "index"=>"microservice-%{+YYYY.MM.dd}"}
18:41:47.400 [[main]-pipeline-manager] ERROR logstash.javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid data stream configuration: ["template", "template_name", "index"]>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:68:in `check_data_stream_config!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:33:in `data_stream_config?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:296:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf", "/usr/share/logstash/pipeline/uptime.conf"], :thread=>"#<Thread:0x98bdee3 run>"}

Option 3) Manually -> Works

First, create a index template manually via API call: PUT _index_template/microservices using /usr/share/logstash/templates/microservices.json

Then:

    output {
      elasticsearch {
        hosts => ["https://elasticsearch-master:9200"]
        index => "microservice-test"
        action => "create"
      }    
    }

But I don't want to do this manual step. I want to use the logstash output options to manage data_stream + index names + index templates.

Answer 1

When using data_stream in your elasticsearch output, you cannot specify any of index, template or template_name since data stream have a specific naming scheme composed of a type, a dataset and a namespace.

In your case, the type seems to be microservice (if not specified it's logs by default), the default dataset is generic and the default namespace is default.

So if your elasticsearch output looks like this...

output {
  elasticsearch {
    hosts => ["https://elasticsearch-master:9200"]
    data_stream => true
    data_stream_type => "microservice"
  }    
}

...your data is going to be stored in a data stream called microservice-generic-default and that's going to match your index template matching microservice-*, so you'd be good to go.

You just need to make sure to create the index template in advance with the following command because Logstash only supports the legacy index templates which don't support data stream

PUT _index_template/microservices
{
  "index_patterns": "microservice-*",
  "template": {
    "settings" : {
        "index" : {
          "number_of_shards" : "1",
          "number_of_replicas" : "1"
        }
      },
    "mappings" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "keyword"
          },
          "host" : {
            "type" : "keyword"
          },
          "level" : {
            "type" : "keyword"
          },
          "service" : {
            "type" : "keyword"
          },
          "type" : {
            "type" : "keyword"
          }
        }
      }
  },
  "data_stream" : {
      "hidden" : false
  }
}

After creating this index template, you can run your Logstash pipeline and it will work.