CODEWITHSUNDEEP
istiokialiistio-kiali
Abhinav Sharma
Abhinav Sharma

Reputation: 27

ERR invalid character '}' looking for beginning of object key string: goroutine 20932 in kiali

I am trying to install Prometheus and Kiali using Istio sample addons. I modifies the file to add nodePort for Service and changed imagePullPolicy to IfNotPresent for Kiali container. The pod comes up, but I am unable to view the graph. It gives the following error:

ERR invalid character '}' looking for beginning of object key string: goroutine 20932

Kiali pod gives the following stack trace:

invalid character '}' looking for beginning of object key string
Additional Detail:
goroutine 2066 [running]:
runtime/debug.Stack(0xc00173aa80, 0x40, 0xc0020d0cc0)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/kiali/kiali/handlers.handlePanic(0x1d5bad0, 0xc001819dc0)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/handlers/graph.go:86 +0x185
panic(0x1841540, 0xc0020d0cc0)
    /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/kiali/kiali/graph.CheckError(...)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/util.go:38
github.com/kiali/kiali/graph/telemetry/istio/appender.ServiceEntryAppender.AppendGraph(0xc000734f90, 0xc0008ae8f7, 0xc, 0xc000735380, 0xc0007352c0, 0xc0020d0a80)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/telemetry/istio/appender/service_entry.go:59 +0x1c6
github.com/kiali/kiali/graph/telemetry/istio.BuildNamespacesTrafficMap(0xc000734f90, 0x0, 0xc0020ed6d0, 0x5, 0x5, 0x100, 0xc000734ea0, 0xdf8475800, 0xc0008ae8f7, 0xc, ...)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/telemetry/istio/istio.go:54 +0x282
github.com/kiali/kiali/graph/api.graphNamespacesIstio(0xc001e625a0, 0xc000d95c00, 0x1af6a6c, 0x9, 0x1af1831, 0x5, 0xc0008ae939, 0x3, 0xdf8475800, 0xc0008ae8f7, ...)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/api/api.go:44 +0xb8
github.com/kiali/kiali/graph/api.GraphNamespaces(0xc001e625a0, 0x1af6a6c, 0x9, 0x1af1831, 0x5, 0xc0008ae939, 0x3, 0xdf8475800, 0xc0008ae8f7, 0xc, ...)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/api/api.go:26 +0x178
github.com/kiali/kiali/handlers.GraphNamespaces(0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/handlers/graph.go:51 +0x13c
net/http.HandlerFunc.ServeHTTP(0x1bc10a0, 0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
    /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/kiali/kiali/routing.metricHandler.func1(0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/routing/router.go:92 +0xa3
net/http.HandlerFunc.ServeHTTP(0xc000132fc0, 0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
    /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/kiali/kiali/handlers.AuthenticationHandler.Handle.func1(0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/handlers/authentication.go:649 +0x1f7
net/http.HandlerFunc.ServeHTTP(0xc000689230, 0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
    /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/kiali/kiali/server.plainHttpMiddleware.func1(0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
    /home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/server/server.go:127 +0x6c
net/http.HandlerFunc.ServeHTTP(0xc00061d740, 0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
    /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00031c0c0, 0x1d5bad0, 0xc001819dc0, 0xc000eec500)
    /home/fedora/workspace/kiali-workflow/kiali-release/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0xd3
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x1d5b9e0, 0xc00075c460, 0xc000eec500)
    /home/fedora/workspace/kiali-workflow/kiali-release/pkg/mod/github.com/!n!y!times/[email protected]/gzip.go:336 +0x22d
net/http.HandlerFunc.ServeHTTP(0xc00025e510, 0x1d5b9e0, 0xc00075c460, 0xc000eec500)
    /usr/local/go/src/net/http/server.go:2069 +0x44
net/http.(*ServeMux).ServeHTTP(0xc00014f800, 0x1d5b9e0, 0xc00075c460, 0xc000eec500)
    /usr/local/go/src/net/http/server.go:2448 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00021c000, 0x1d5b9e0, 0xc00075c460, 0xc000eec500)
    /usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc0002c65a0, 0x1d5f6b8, 0xc00131fa40)
    /usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3013 +0x39b

Version details:

Edit: Adding manifest files used for deployment.

Kiali Configmap:

# Source: kiali-server/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali
  namespace: istio-system
  labels:
    helm.sh/chart: kiali-server-1.34.0
    app: kiali
    app.kubernetes.io/name: kiali
    app.kubernetes.io/instance: kiali-server
    version: "v1.34.0"
    app.kubernetes.io/version: "v1.34.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: "kiali"
data:
  config.yaml: |
    auth:
      openid: {}
      openshift:
        client_id_prefix: kiali
      strategy: anonymous
    deployment:
      accessible_namespaces:
      - '**'
      additional_service_yaml: {}
      affinity:
        node: {}
        pod: {}
        pod_anti: {}
      custom_dashboards:
        excludes:
        - ""
        includes:
        - '*'
      hpa:
        api_version: autoscaling/v2beta2
        spec: {}
      image_name: quay.io/kiali/kiali
      image_pull_policy: IfNotPresent
      image_pull_secrets: []
      image_version: v1.34
      ingress_enabled: false
      logger:
        log_format: text
        log_level: info
        sampler_rate: "1"
        time_field_format: 2006-01-02T15:04:05Z07:00
      namespace: istio-system
      node_selector: {}
      override_ingress_yaml:
        metadata: {}
      pod_annotations:
        sidecar.istio.io/inject: "false"
      pod_labels: {}
      priority_class_name: ""
      replicas: 1
      resources: {}
      secret_name: kiali
      service_annotations: {}
      service_type: ""
      tolerations: []
      version_label: v1.34.0
      view_only_mode: false
    external_services:
      custom_dashboards:
        enabled: true
    identity:
      cert_file: ""
      private_key_file: ""
    istio_namespace: istio-system
    login_token:
      signing_key: CHANGEME
    server:
      metrics_enabled: true
      metrics_port: 9090
      port: 20001
      web_root: /kiali
---
# Source: kiali-server/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kiali
  namespace: istio-system
  labels:
    helm.sh/chart: kiali-server-1.34.0
    app: kiali
    app.kubernetes.io/name: kiali
    app.kubernetes.io/instance: kiali-server
    version: "v1.34.0"
    app.kubernetes.io/version: "v1.34.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: "kiali"
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kiali
      app.kubernetes.io/instance: kiali-server
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      name: kiali
      labels:
        helm.sh/chart: kiali-server-1.34.0
        app: kiali
        app.kubernetes.io/name: kiali
        app.kubernetes.io/instance: kiali-server
        version: "v1.34.0"
        app.kubernetes.io/version: "v1.34.0"
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: "kiali"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        kiali.io/runtimes: go,kiali
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: kiali
      containers:
      - image: "quay.io/kiali/kiali:v1.34"
        imagePullPolicy: IfNotPresent
        name: kiali
        command:
        - "/opt/kiali/kiali"
        - "-config"
        - "/kiali-configuration/config.yaml"
        ports:
        - name: api-port
          containerPort: 20001
        - name: http-metrics
          containerPort: 9090
        readinessProbe:
          httpGet:
            path: /kiali/healthz
            port: api-port
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 30
        livenessProbe:
          httpGet:
            path: /kiali/healthz
            port: api-port
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 30
        env:
        - name: ACTIVE_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: LOG_LEVEL
          value: "info"
        - name: LOG_FORMAT
          value: "text"
        - name: LOG_TIME_FIELD_FORMAT
          value: "2006-01-02T15:04:05Z07:00" 
        - name: LOG_SAMPLER_RATE
          value: "1"
        volumeMounts:
        - name: kiali-configuration
          mountPath: "/kiali-configuration"
        - name: kiali-cert
          mountPath: "/kiali-cert"
        - name: kiali-secret
          mountPath: "/kiali-secret"
      volumes:
      - name: kiali-configuration
        configMap:
          name: kiali
      - name: kiali-cert
        secret:
          secretName: istio.kiali-service-account
          optional: true
      - name: kiali-secret
        secret:
          secretName: kiali
          optional: true
---
# Source: kiali-server/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kiali
  namespace: istio-system
  labels:
    helm.sh/chart: kiali-server-1.34.0
    app: kiali
    app.kubernetes.io/name: kiali
    app.kubernetes.io/instance: kiali-server
    version: "v1.34.0"
    app.kubernetes.io/version: "v1.34.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: "kiali"
  annotations:
spec:
  ports:
  - name: http
    protocol: TCP
    port: 20001
    nodePort: 31334
  - name: http-metrics
    protocol: TCP
    port: 9090
    nodePort: 31335
  selector:
    app.kubernetes.io/name: kiali
    app.kubernetes.io/instance: kiali-server
  type: LoadBalancer

Prometheus:

# Source: prometheus/templates/server/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    component: "server"
    app: prometheus
    release: prometheus
    chart: prometheus-13.6.0
    heritage: Helm
  name: prometheus
  namespace: istio-system
data:
  alerting_rules.yml: |
    groups:
    - name: Alerts
      rules:
      - alert: High Pod Memory
        expr: sum(container_memory_usage_bytes) > 1
        for: 5m
        labels:
          severity: slack
        annotations:
          summary: High Memory Usage
  alerts: |
    {}
  prometheus.yml: |
    global:
      evaluation_interval: 1m
      scrape_interval: 15s
      scrape_timeout: 10s
    rule_files:
    - /etc/config/recording_rules.yml
    - /etc/config/alerting_rules.yml
    - /etc/config/rules
    - /etc/config/alerts
    alerting:
      alertmanagers:
        - scheme: http
          static_configs:
            - targets:
              - "alertmanager:9093"
    scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets:
        - localhost:9090
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-apiservers
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: default;kubernetes;https
        source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_service_name
        - __meta_kubernetes_endpoint_port_name
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - job_name: kubernetes-service-endpoints
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: kubernetes_name
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: kubernetes_node
    - job_name: kubernetes-service-endpoints-slow
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: kubernetes_name
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: kubernetes_node
      scrape_interval: 5m
      scrape_timeout: 30s
    - honor_labels: true
      job_name: prometheus-pushgateway
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - action: keep
        regex: pushgateway
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
    - job_name: kubernetes-services
      kubernetes_sd_configs:
      - role: service
      metrics_path: /probe
      params:
        module:
        - http_2xx
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
      - source_labels:
        - __address__
        target_label: __param_target
      - replacement: blackbox
        target_label: __address__
      - source_labels:
        - __param_target
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - source_labels:
        - __meta_kubernetes_service_name
        target_label: kubernetes_name
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: kubernetes_pod_name
      - action: drop
        regex: Pending|Succeeded|Failed
        source_labels:
        - __meta_kubernetes_pod_phase
    - job_name: kubernetes-pods-slow
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: kubernetes_pod_name
      - action: drop
        regex: Pending|Succeeded|Failed
        source_labels:
        - __meta_kubernetes_pod_phase
      scrape_interval: 5m
      scrape_timeout: 30s
  recording_rules.yml: |
    {}
  rules: |
    {}
---
# Source: prometheus/templates/server/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: "server"
    app: prometheus
    release: prometheus
    chart: prometheus-13.6.0
    heritage: Helm
  name: prometheus
  namespace: istio-system
spec:
  selector:
    matchLabels:
      component: "server"
      app: prometheus
      release: prometheus
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        component: "server"
        app: prometheus
        release: prometheus
        chart: prometheus-13.6.0
        heritage: Helm
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus-server-configmap-reload
          image: "jimmidyson/configmap-reload:v0.5.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://127.0.0.1:9090/-/reload
          resources:
            {}
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
              readOnly: true

        - name: prometheus-server
          image: "prom/prometheus:v2.24.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - --storage.tsdb.retention.time=15d
            - --config.file=/etc/config/prometheus.yml
            - --storage.tsdb.path=/data
            - --web.console.libraries=/etc/prometheus/console_libraries
            - --web.console.templates=/etc/prometheus/consoles
            - --web.enable-lifecycle
          ports:
            - containerPort: 9090
              #readinessProbe:
              #httpGet:
              #path: /-/ready
              #port: 9090
              #initialDelaySeconds: 0
              #periodSeconds: 5
              #timeoutSeconds: 4
              #failureThreshold: 3
              #successThreshold: 1
              #livenessProbe:
              #httpGet:
              #path: /-/healthy
              #port: 9090
              #initialDelaySeconds: 30
              #periodSeconds: 15
              #timeoutSeconds: 10
              #failureThreshold: 3
              #successThreshold: 1
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
            - name: storage-volume
              mountPath: /data
              subPath: ""
      hostNetwork: false
      dnsPolicy: ClusterFirst
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      terminationGracePeriodSeconds: 300
      volumes:
        - name: config-volume
          configMap:
            name: prometheus
        - name: storage-volume
          emptyDir:
            {}
---
# Source: prometheus/templates/server/service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    component: "server"
    app: prometheus
    release: prometheus
    chart: prometheus-13.6.0
    heritage: Helm
  name: prometheus
  namespace: istio-system
spec:
  ports:
    - name: http
      port: 9090
      protocol: TCP
      targetPort: 9090
      nodePort: 31333
  selector:
    component: "server"
    app: prometheus
    release: prometheus
  sessionAffinity: None
  type: LoadBalancer

Upvotes: 1

Views: 574

Answers (1)

user15659347
user15659347

Reputation:

If there is no particular reason for LoadBalancer I would recommend reverting those manifests back to the original, provided with Istio.

If you want to access dashboard you can do so by issuing

istioctl dashboard kiali

for Kiali, or

istioctl dashboard prometheus

for Prometheus.

Optionally you can expose those dashboards using Istio Gateway and VirtualService. More details are available here.

However, I would not recommend exposing Kiali and Prometheus to outside, as it creates a huge security risk.

Upvotes: 0

Related Questions