Kibana - Request Timeout after 30000ms at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15

Question

Fresh installment of Kibana (On redhat 7.6 (64bit) via yum) starts, but is restarting every minute. Before it restarted every 5 seconds, but i fixed it after changing /etc/fstab to allow noexec on /var cause it is needed for /var/lib/kibana/headless_shell-linux/headless_shell.

I tried starting Kibana by command instead of systemctl to see full logs: /usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml

It gives me output:

  log   [14:59:49.784] [info][status][plugin:kibana@undefined] Status changed from uninitialized to green - Ready 
  log   [14:59:49.812] [info][status][plugin:elasticsearch@undefined] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.814] [info][status][plugin:xpack_main@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.820] [info][status][plugin:graph@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.828] [info][status][plugin:monitoring@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:49.831] [info][status][plugin:spaces@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.841] [warning][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml 
  log   [14:59:49.845] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended. 
  log   [14:59:49.851] [info][status][plugin:security@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.872] [info][status][plugin:searchprofiler@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.874] [info][status][plugin:ml@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.907] [info][status][plugin:tilemap@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.909] [info][status][plugin:watcher@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.922] [info][status][plugin:grokdebugger@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.926] [info][status][plugin:dashboard_mode@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:49.927] [info][status][plugin:logstash@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.932] [info][status][plugin:beats_management@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.945] [info][status][plugin:apm_oss@undefined] Status changed from uninitialized to green - Ready 
  log   [14:59:49.956] [info][status][plugin:apm@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:49.957] [info][status][plugin:tile_map@undefined] Status changed from uninitialized to green - Ready 
  log   [14:59:49.959] [info][status][plugin:task_manager@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:49.961] [info][status][plugin:maps@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.965] [info][status][plugin:interpreter@undefined] Status changed from uninitialized to green - Ready 
  log   [14:59:49.972] [info][status][plugin:canvas@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:49.975] [info][status][plugin:license_management@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:49.977] [info][status][plugin:cloud@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:49.978] [info][status][plugin:index_management@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:49.999] [info][status][plugin:console@undefined] Status changed from uninitialized to green - Ready 
  log   [14:59:50.002] [info][status][plugin:console_extensions@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:50.004] [info][status][plugin:notifications@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:50.006] [info][status][plugin:index_lifecycle_management@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:50.039] [info][status][plugin:infra@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:50.041] [info][status][plugin:rollup@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:50.048] [info][status][plugin:remote_clusters@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:50.053] [info][status][plugin:cross_cluster_replication@7.1.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch 
  log   [14:59:50.061] [info][status][plugin:translations@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:50.069] [info][status][plugin:upgrade_assistant@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:50.084] [info][status][plugin:uptime@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:50.086] [info][status][plugin:oss_telemetry@7.1.1] Status changed from uninitialized to green - Ready 
  log   [14:59:50.094] [info][status][plugin:metrics@undefined] Status changed from uninitialized to green - Ready 
  log   [14:59:50.210] [info][status][plugin:timelion@undefined] Status changed from uninitialized to green - Ready 
  log   [14:59:50.507] [info][status][plugin:elasticsearch@undefined] Status changed from yellow to green - Ready 
  log   [14:59:50.513] [error][status][plugin:xpack_main@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.514] [error][status][plugin:graph@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.514] [error][status][plugin:spaces@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.514] [error][status][plugin:security@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.515] [error][status][plugin:searchprofiler@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.515] [error][status][plugin:ml@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.515] [error][status][plugin:tilemap@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.515] [error][status][plugin:watcher@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.516] [error][status][plugin:grokdebugger@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.516] [error][status][plugin:logstash@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.516] [error][status][plugin:beats_management@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.516] [error][status][plugin:maps@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.517] [error][status][plugin:index_management@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.517] [error][status][plugin:index_lifecycle_management@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.517] [error][status][plugin:rollup@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.517] [error][status][plugin:remote_clusters@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.517] [error][status][plugin:cross_cluster_replication@7.1.1] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information. 
  log   [14:59:50.901] [warning][browser-driver][reporting] Enabling the Chromium sandbox provides an additional layer of protection. 
  log   [14:59:50.903] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml 
  log   [14:59:50.919] [error][status][plugin:reporting@7.1.1] Status changed from uninitialized to red - [data] Elasticsearch cluster did not respond with license information. 
 error  [15:00:20.514] [warning][process] UnhandledPromiseRejectionWarning: Error: Request Timeout after 30000ms 
    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15 
    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7) 
    at ontimeout (timers.js:436:11) 
    at tryOnTimeout (timers.js:300:5) 
    at listOnTimeout (timers.js:263:5) 
    at Timer.processTimers (timers.js:223:10) 
    at emitWarning (internal/process/promises.js:81:15) 
    at emitPromiseRejectionWarnings (internal/process/promises.js:120:9) 
    at process._tickCallback (internal/process/next_tick.js:69:34) 
 error  [15:00:20.515] [warning][process] Error: Request Timeout after 30000ms 
    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15 
    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7) 
    at ontimeout (timers.js:436:11) 
    at tryOnTimeout (timers.js:300:5) 
    at listOnTimeout (timers.js:263:5) 
    at Timer.processTimers (timers.js:223:10) 
  log   [15:00:20.908] [warning][reporting] Could not retrieve cluster settings, because of Request Timeout after 30000ms 
  log   [15:00:20.940] [warning][task_manager] PollError Request Timeout after 30000ms 
  log   [15:00:20.942] [warning][maps] Error scheduling telemetry task, received NotInitialized: Tasks cannot be scheduled until after task manager is initialized! 
  log   [15:00:20.943] [warning][telemetry] Error scheduling task, received NotInitialized: Tasks cannot be scheduled until after task manager is initialized! 

I want Kibana to start without failing.

Kibana.yml:

    server.port: 5601
    server.host: "xxx.xx.xx.x"
    server.name: "elk-log-kibana"
    elasticsearch.hosts: "http://localhost:9200"
    server.basePath: "/kibana"
    server.rewriteBasePath: true

elasticsearch.yml:

    cluster.name: elk-log-elasticsearch    
    path.data: /var/lib/elasticsearch
    path.logs: /var/log/elasticsearch
    http.port: 9200
    network.host: 0.0.0.0
    discovery.seed_hosts: 127.0.0.1

Elasticsearch works fine:

curl -v http://localhost:9200 
* About to connect() to localhost port 9200 (#0) 
*   Trying ::1... 
* Connected to localhost (::1) port 9200 (#0) 
> GET / HTTP/1.1 
> User-Agent: curl/7.29.0 
> Host: localhost:9200 
> Accept: */* 
> 
< HTTP/1.1 200 OK 

Answer 1

I am very new to the elasticsearch technology, just exploring and came across this thing while configuring my kibana.

I Don't know whether know about this or not but just reminding you about configuration file of kibana that is kibana.yml in your config folder in which you will find below statements

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

you can change it according to your requirement, this might help you. correct me if i am wrong.