CODEWITHSUNDEEP
dockerelasticsearchkibana
pantonis
pantonis

Reputation: 6517

Elastic with Kibana on docker gives Kibana server is not ready yet

I have Elastic & Kibana and suddenly last week it stopped working with the error Kibana server is not ready yet. I have seen numerous threads on gooogle about this problem. Tried it all with no luck at all. I can see in the kibana logs the following errors:

License information could not be obtained from Elasticsearch due to Error: Error: certificate has expired error"}
{"type":"log","@timestamp":"2020-05-31T13:45:24Z","tags":["info","monitoring","kibana-monitoring"],"pid":6,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2020-05-31T13:45:54Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://el.mydomain.io:9200/_xpack => certificate has expired"}

I checked the certificate. It is not expired. It expires next February. Any idea what is happening?

Update #1: Added docker-compose.yml

version: '2.2'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: el.mydomain.io
    environment:
      - node.name=el.mydomain.io
      - discovery.type=single-node       
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"      
      - xpack.license.self_generated.type=trial # <1>
      - xpack.security.enabled=true      
      - xpack.security.http.ssl.enabled=true # <2>
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true # <3>
      - xpack.security.transport.ssl.verification_mode=certificate # <4>
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes: 
      - data01:/usr/share/elasticsearch/data
      - /home/user11/SSL-Certs/:$CERTS_DIR
    ports:
      - 9200:9200
    networks:
      - elastic

    healthcheck:
      test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://el.mydomain.io:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib.mydomain.io
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 443:5601    
    environment:
      SERVERNAME: kib.mydomain.io
      SERVER.HOST: kib.mydomain.io
      ELASTICSEARCH_URL: https://el.mydomain.io:9200
      ELASTICSEARCH_HOSTS: https://el.mydomain.io:9200
      ELASTICSEARCH_USERNAME: kibana
      ELASTICSEARCH_PASSWORD: Mypassword
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
    volumes: 
      - /home/user11/SSL-Certs/:$CERTS_DIR
    networks:
      - elastic    
volumes:
  data01:
    driver: local
    #certs:
    #driver: local

networks: 
  elastic:
    driver: bridge

Update #2: Output from certificates:

C:\Temp>openssl verify -CAfile ca.crt es01.crt
C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
error 10 at 3 depth lookup: certificate has expired
C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
error 10 at 2 depth lookup: certificate has expired
error es01.crt: verification failed

C:\Temp>openssl verify -CAfile ca.crt kib01.crt
C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
error 10 at 3 depth lookup: certificate has expired
C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
error 10 at 2 depth lookup: certificate has expired
error kib01.crt: verification failed

C:\Temp>openssl x509 -noout -in es01.crt -dates
notBefore=Jan  2 00:00:00 2020 GMT
notAfter=Jan  1 23:59:59 2021 GMT

C:\Temp>openssl x509 -noout -in kib01.crt -dates
notBefore=Jan  2 00:00:00 2020 GMT
notAfter=Jan  1 23:59:59 2021 GMT

C:\Temp>openssl x509 -noout -in ca.crt -dates
notBefore=Nov  2 00:00:00 2018 GMT
notAfter=Dec 31 23:59:59 2030 GMT

Upvotes: 0

Views: 1547

Answers (2)

Val
Val

Reputation: 217594

It's not a license issue, but your SSL certificate has expired. The error message states

Request error, retrying\nGET https://el.mydomain.io:9200/_xpack => certificate has expired

So you simply need to renew your certificate

Upvotes: 1

user11935734
user11935734

Reputation:

looking at this part of your error message

{"type":"log","@timestamp":"2020-05-31T13:45:54Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://el.mydomain.io:9200/_xpack => certificate has expired"}

Looks like your x-pack license expired which might be causing the issue, can you please fix this issue or remove the x-pack and try again.

Upvotes: 1

Related Questions