CODEWITHSUNDEEP
elasticsearchkubernetesamazon-eks
Anshul Tripathi
Anshul Tripathi

Reputation: 599

Elasticsearch fails to start on AWS kubernetes cluster

I am running my kubernetes cluster on AWS EKS which runs kubernetes 1.10. I am following this guide to deploy elasticsearch in my Cluster elasticsearch Kubernetes

The first time I deployed it everything worked fine. Now, When I redeploy it gives me the following error.

ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2018-08-24T18:07:28,448][INFO ][o.e.n.Node               ] [es-master-6987757898-5pzz9] stopping ...
[2018-08-24T18:07:28,534][INFO ][o.e.n.Node               ] [es-master-6987757898-5pzz9] stopped
[2018-08-24T18:07:28,534][INFO ][o.e.n.Node               ] [es-master-6987757898-5pzz9] closing ...
[2018-08-24T18:07:28,555][INFO ][o.e.n.Node               ] [es-master-6987757898-5pzz9] closed

Here is my deployment file.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: es-master
  labels:
    component: elasticsearch
    role: master
spec:
  replicas: 3
  template:
    metadata:
      labels:
        component: elasticsearch
        role: master
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox:1.27.2
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      containers:
      - name: es-master
        image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: CLUSTER_NAME
          value: myesdb
        - name: NUMBER_OF_MASTERS
          value: "2"
        - name: NODE_MASTER
          value: "true"
        - name: NODE_INGEST
          value: "false"
        - name: NODE_DATA
          value: "false"
        - name: HTTP_ENABLE
          value: "false"
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        - name: NETWORK_HOST
          value: "0.0.0.0"
        - name: PROCESSORS
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
        resources:
          requests:
            cpu: 0.25
          limits:
            cpu: 1
        ports:
        - containerPort: 9300
          name: transport
        livenessProbe:
          tcpSocket:
            port: transport
          initialDelaySeconds: 20
          periodSeconds: 10
        volumeMounts:
        - name: storage
          mountPath: /data
      volumes:
          - emptyDir:
              medium: ""
            name: "storage"

I have seen a lot of posts talking about increasing the value but I am not sure how to do it. Any help would be appreciated.

Upvotes: 5

Views: 2109

Answers (3)

Preston
Preston

Reputation: 163

This is the only thing that worked for me using EKS setting up an EFK stack. Add this to your nodegroup creation YAML file under nodeGroups:. Then create your nodegroup and apply your ES pods on it.

preBootstrapCommands:
  - "sysctl -w  vm.max_map_count=262144"
  - "systemctl restart docker"

Upvotes: 0

ozlevka
ozlevka

Reputation: 2146

Just want to append to this issue:

If you create EKS cluster by eksctl then you can append to NodeGroup creation yaml:

 preBootstrapCommand:
      - "sed -i -e 's/1024:4096/65536:65536/g' /etc/sysconfig/docker"
      - "systemctl restart docker"

This will solve the problem for newly created cluster by fixing docker daemon config.

Upvotes: 1

Alexsey Nadtochey
Alexsey Nadtochey

Reputation: 1

Update default-ulimit parameter in the file '/etc/docker/daemon.json'

  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Soft": 65536,
      "Hard": 65536
    }
  }

and restart docker daemon.

Upvotes: 0

Related Questions