kkoziarski
kkoziarski

Reputation: 121

How to configure Elasticsearch Index Lifecycle Management (ILM) durring installation in YAML file

I would like to configure default Index Lifecycle Management (ILM) policy and index template durring installation ES in kubernetes cluster, in the YAML installation file, instead of calling ES API after installation. How can I do that?

I have Elasticsearch installed in kubernetes cluster based on YAML file.

The following works queries work.

PUT _ilm/policy/logstash_policy
{
    "policy": {
        "phases": {
            "delete": {
                "min_age": "30d",
                "actions": {
                    "delete": {}
                }
            }
        }
    }
}
PUT _template/logstash_template
{
    "index_patterns": ["logstash-*"],
    "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1,
        "index.lifecycle.name": "logstash_policy"
    }
}

I would like to have above setup just after installation, without making any curl queries.

Upvotes: 4

Views: 4563

Answers (2)

Daniel Paulus
Daniel Paulus

Reputation: 131

I've used the answer to get a custom policy in place for Packetbeat running with ECK.

The ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: packetbeat-ilmpolicy
  labels:
    k8s-app: packetbeat
data:
  ilm-policy.json: |-
    {
      "policy": {
        "phases": {
          "hot": {
            "min_age": "0ms",
            "actions": {
              "rollover": {
                "max_age": "1d"
              }
            }
          },
          "delete": {
            "min_age": "1d",
            "actions": {
                "delete": {}
            }
          }
        }
      }
    }

The Beat config:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: packetbeat
spec:
  type: packetbeat
  elasticsearchRef:
    name: demo
  kibanaRef:
    name: demo
  config:
    pipeline: geoip-info
    packetbeat.interfaces.device: any
    packetbeat.protocols:
      - type: dns
        ports: [53]
        include_authorities: true
        include_additionals: true
      - type: http
        ports: [80, 8000, 8080, 9200, 9300]
      - type: tls
        ports: [443, 993, 995, 5223, 8443, 8883, 9243]
    packetbeat.flows:
      timeout: 30s
      period: 30s
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}
    setup.ilm:
      enabled: true
      overwrite: true
      policy_name: "packetbeat"
      policy_file: /usr/share/packetbeat/ilm-policy.json
      pattern: "{now/d}-000001"
  daemonSet:
    podTemplate:
      spec:
        terminationGracePeriodSeconds: 30
        hostNetwork: true
        automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
        dnsPolicy: ClusterFirstWithHostNet
        tolerations:
          - operator: Exists
        containers:
          - name: packetbeat
            securityContext:
              runAsUser: 0
              capabilities:
                add:
                  - NET_ADMIN
            volumeMounts:
              - name: ilmpolicy-config
                mountPath: /usr/share/packetbeat/ilm-policy.json
                subPath: ilm-policy.json
                readOnly: true
        volumes:
          - name: ilmpolicy-config
            configMap:
              name: packetbeat-ilmpolicy

The important parts in the Beat config are the Volume mount where we mount the configmap into the container.

After this we can reference the file in the config with setup.ilm.policy_file.

Upvotes: 2

Ludo
Ludo

Reputation: 109

I'll try to answer both of your questions.

index template

You can pass the index template with this configuration in your elasticsearch yaml. For instance:

setup.template:
  name: "<chosen template name>-%{[agent.version]}"
  pattern: "<chosen pattern name>-%{[agent.version]}-*"

Checkout the ES documentation to see where exactly this setup.template belongs and you're good to go.

ilm policy

The way to make this work is to get the ilm-policy.json file that has your ilm configuration to the pod's /usr/share/filebeat/ directory. in your YAML installation file, you can then use this line in your config to get it to work (I've added my whole ilm config):

setup.ilm:
  enabled: true
  policy_name: "<policy name>"
  rollover_alias: "<rollover alias name
  policy_file: "ilm-policy.json"
  pattern: "{now/d}-000001"

So, how to get the file there? The ingredients are 1 configmap containing your ilm-policy.json, and a volume and volumeMount in your daemonset configuration to mount the configmap's contents to the pod's directories.

Note: I used helm for deploying filebeat to an AKS cluster (v 1.15), which connects to Elastic cloud. In your case, the application folder to store your json will probably be /usr/share/elasticsearch/ilm-policy.json.

Below, you'll see a line like {{ .Files.Get <...> }}, which is a templating function for helm getting the contents of the files. Alternatively, you can copy the file contents directly into the configmap yaml, but to have the file separate makes it better managable in my opinion.

The configMap

Make sure your ilm-policy.json is somewhere reachable by your deployments. This is how the configmap can look:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ilmpolicy-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  ilm-policy.json: |-
{{ .Files.Get "ilm-policy.json" | indent 4 }}

The Daemonset

at the deamonSet's volumeMounts section, append this:

- name: ilm-configmap-volume
  mountPath: /usr/share/filebeat/ilm-policy.json
  subPath: ilm-policy.json
  readOnly: true

and at the volume section append this:

 - name: ilm-configmap-volume
   configMap:
     name: ilmpolicy-config

I'm not exactly sure the spacing is correct in the browser, but this should give a pretty good idea. I hope this works for your setup! good luck.

Upvotes: 5

Related Questions