CODEWITHSUNDEEP
kubernetescluster-computinggrafanamonitoringamazon-eks
Tom Burton
Tom Burton

Reputation: 51

Grafana continuously reports 401 errors attempting to access the endpoint /api/ws/live

We are running Grafana on EKS Kubernetes v1.21 as a Helm deployment behind a Traefik reverse proxy.

Grafana version: v9.0.3

Recently, Grafana has been posting this same log message every minute without fail:

2022-08-24 15:52:47 
logger=context traceID=00000000000000000000000000000000 userId=0 orgId=0 uname= t=2022-08-24T13:52:47.293094029Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.1.3.153 time_ms=4 duration=4.609805ms size=27 referer= traceID=00000000000000000000000000000000
2022-08-24 15:52:47 
logger=context traceID=00000000000000000000000000000000 t=2022-08-24T13:52:47.290478899Z level=error msg="Failed to look up user based on cookie" error="user token not found"

I can't confirm whether these two log messages are related but I believe they are.

I cannot find any user with id 0.

Another log error I see occasionally is

2022-08-24 15:43:43 
logger=ngalert t=2022-08-24T13:43:43.020553296Z level=error msg="unable to fetch orgIds" msg="context canceled"

What I can see, is that the remote_addr refers to the node in our cluster that Grafana is deployed on.

Can anyone explain why this is continually hitting the endpoint shown?

Thanks!

Upvotes: 5

Views: 4859

Answers (1)

Daniel Lee
Daniel Lee

Reputation: 7969

The Grafana Live feature is real-time messaging that uses websockets. It is used in Grafana for notifying on events like someone else is editing the same dashboard as you. It can also be used for streaming data directly to Grafana. Docs here

You can either turn off Grafana Live or configure your proxy to allow websockets.

Upvotes: 1

Related Questions