Kibana server is not ready yet - [security_exception] unable to authenticate user [elastic]

Question

What happen is I tried to add user for ElasticSearch and Kibana. For ElasticSearch, I added xpack.security.enabled: true at elasticsearch.yml and elasticsearch.username: "elastic" and elasticsearch.password: "ipF2vorNqvRgXTjuptqS" in kibana.yml.

When I start ElasticSearch, I was prompted to key in username and password. I did so and successfully login.

But when I start Kibana, in the log I get this error: [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [elastic] for REST request [/_xpack]

At http://localhost:5601, I got this error Kibana server is not ready yet

To troubleshoot, I run http://localhost:9200/_security/user/ and I get

{
   "elastic":{
      "username":"elastic",
      "roles":[
         "superuser"
      ],
      "full_name":null,
      "email":null,
      "metadata":{
         "_reserved":true
      },
      "enabled":true
   },
   "kibana":{
      "username":"kibana",
      "roles":[
         "kibana_system"
      ],
      "full_name":null,
      "email":null,
      "metadata":{
         "_deprecated":true,
         "_deprecated_reason":"Please use the [kibana_system] user instead.",
         "_reserved":true
      },
      "enabled":true
   },
   "kibana_system":{
      "username":"kibana_system",
      "roles":[
         "kibana_system"
      ],
      "full_name":null,
      "email":null,
      "metadata":{
         "_reserved":true
      },
      "enabled":true
   },
   "logstash_system":{
      "username":"logstash_system",
      "roles":[
         "logstash_system"
      ],
      "full_name":null,
      "email":null,
      "metadata":{
         "_reserved":true
      },
      "enabled":true
   },
   "beats_system":{
      "username":"beats_system",
      "roles":[
         "beats_system"
      ],
      "full_name":null,
      "email":null,
      "metadata":{
         "_reserved":true
      },
      "enabled":true
   },
   "apm_system":{
      "username":"apm_system",
      "roles":[
         "apm_system"
      ],
      "full_name":null,
      "email":null,
      "metadata":{
         "_reserved":true
      },
      "enabled":true
   },
   "remote_monitoring_user":{
      "username":"remote_monitoring_user",
      "roles":[
         "remote_monitoring_collector",
         "remote_monitoring_agent"
      ],
      "full_name":null,
      "email":null,
      "metadata":{
         "_reserved":true
      },
      "enabled":true
   }
}

I follow the steps to setup my password at elasticsearch.

D:\elasticsearch\bin>elasticsearch-setup-passwords auto
future versions of Elasticsearch will require Java 11; your Java version from [C:\Program Files\Java\jdk1.8.0_251\jre] does not meet this requirement
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y


Changed password for user apm_system
PASSWORD apm_system = TCxggBZ1O8u7pCYMQZx3

Changed password for user kibana_system
PASSWORD kibana_system = G48r4h6M6WjLnjzPqjAG

Changed password for user kibana
PASSWORD kibana = G48r4h6M6WjLnjzPqjAG

Changed password for user logstash_system
PASSWORD logstash_system = UQZTsQrN84jQuzCKnOSc

Changed password for user beats_system
PASSWORD beats_system = wC5h5tShmOuouJ072owM

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = VHqOCfKuxbCCjbEMTWQZ

Changed password for user elastic
PASSWORD elastic = ipF2vorNqvRgXTjuptqS

How do I troubleshoot this further or solve this? Should I use "elastic" or "kibana_system" as username in kibana.yml?

This is kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "elastic"
elasticsearch.password: "ipF2vorNqvRgXTjuptqS"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

#elasticsearch.username: "elastic"
#elasticsearch.password: "ipF2vorNqvRgXTjuptqS"

#elasticsearch.username: "kibana_system"
#elasticsearch.password: "kibanapassword"

Answer 1

Go to config > kibana.yml in kibana folder.

Change this

elasticsearch.hosts: ["https://172.35.92.97:9200"] 

to

elasticsearch.hosts: ["http://172.35.92.97:9200"].

Make comment followings :

elasticsearch.ssl.certificateAuthorities: [/Users/ozgurhanpolat/Downloads/kibana-8.11.1/data/ca_1704372255930.crt]

and

xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://172.35.92.97:9200'], ca_trusted_fingerprint: b2aed3603d21e57e369af8e9960f94b20a7d93f6b91aac219c908fdd189d737f}] .

Save it and close kibana and run it again.

Don't forget you have to configure kibana and run at least one time before above.

Note that This works in kibana 8.11.1 and elasticsearch 8.11.1 versions.

Answer 2

You need to configure the password too in kibana.yml file: elasticsearch.password:$password

Check the docs here: https://www.elastic.co/guide/en/kibana/current/using-kibana-with-security.html