Reputation: 6517
I have Elastic & Kibana and suddenly last week it stopped working with the error Kibana server is not ready yet
. I have seen numerous threads on gooogle about this problem. Tried it all with no luck at all.
I can see in the kibana logs the following errors:
License information could not be obtained from Elasticsearch due to Error: Error: certificate has expired error"}
{"type":"log","@timestamp":"2020-05-31T13:45:24Z","tags":["info","monitoring","kibana-monitoring"],"pid":6,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2020-05-31T13:45:54Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://el.mydomain.io:9200/_xpack => certificate has expired"}
I checked the certificate. It is not expired. It expires next February. Any idea what is happening?
Update #1: Added docker-compose.yml
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
container_name: el.mydomain.io
environment:
- node.name=el.mydomain.io
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.license.self_generated.type=trial # <1>
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true # <2>
- xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.enabled=true # <3>
- xpack.security.transport.ssl.verification_mode=certificate # <4>
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
- /home/user11/SSL-Certs/:$CERTS_DIR
ports:
- 9200:9200
networks:
- elastic
healthcheck:
test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://el.mydomain.io:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
interval: 30s
timeout: 10s
retries: 5
kib01:
image: docker.elastic.co/kibana/kibana:${VERSION}
container_name: kib.mydomain.io
depends_on: {"es01": {"condition": "service_healthy"}}
ports:
- 443:5601
environment:
SERVERNAME: kib.mydomain.io
SERVER.HOST: kib.mydomain.io
ELASTICSEARCH_URL: https://el.mydomain.io:9200
ELASTICSEARCH_HOSTS: https://el.mydomain.io:9200
ELASTICSEARCH_USERNAME: kibana
ELASTICSEARCH_PASSWORD: Mypassword
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
SERVER_SSL_ENABLED: "true"
SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
volumes:
- /home/user11/SSL-Certs/:$CERTS_DIR
networks:
- elastic
volumes:
data01:
driver: local
#certs:
#driver: local
networks:
elastic:
driver: bridge
Update #2: Output from certificates:
C:\Temp>openssl verify -CAfile ca.crt es01.crt
C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
error 10 at 3 depth lookup: certificate has expired
C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
error 10 at 2 depth lookup: certificate has expired
error es01.crt: verification failed
C:\Temp>openssl verify -CAfile ca.crt kib01.crt
C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
error 10 at 3 depth lookup: certificate has expired
C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
error 10 at 2 depth lookup: certificate has expired
error kib01.crt: verification failed
C:\Temp>openssl x509 -noout -in es01.crt -dates
notBefore=Jan 2 00:00:00 2020 GMT
notAfter=Jan 1 23:59:59 2021 GMT
C:\Temp>openssl x509 -noout -in kib01.crt -dates
notBefore=Jan 2 00:00:00 2020 GMT
notAfter=Jan 1 23:59:59 2021 GMT
C:\Temp>openssl x509 -noout -in ca.crt -dates
notBefore=Nov 2 00:00:00 2018 GMT
notAfter=Dec 31 23:59:59 2030 GMT
Upvotes: 0
Views: 1547
Reputation: 217594
It's not a license issue, but your SSL certificate has expired. The error message states
Request error, retrying\nGET https://el.mydomain.io:9200/_xpack => certificate has expired
So you simply need to renew your certificate
Upvotes: 1
Reputation:
looking at this part of your error message
{"type":"log","@timestamp":"2020-05-31T13:45:54Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://el.mydomain.io:9200/_xpack => certificate has expired"}
Looks like your x-pack license expired which might be causing the issue, can you please fix this issue or remove the x-pack and try again.
Upvotes: 1