Jaeger is not showing any trace results

Question

I am able to run kiali fine. But it Jaeger is not showing any results. I'm using virtualbox for this exercise. In order for me to view it in my local browser I'm using port forwarding.

I think this is communication issue between pods.

Below is what I'm using.

Virtualbox
Minimal install of CentOS_8.4.2105
istio-1.11.4
Docker version 20.10.9, build c2ea9bc
minikube version: v1.23.2
[centos@centos8 bin]$ kubectl version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:32:41Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

here is my Kiali

Below is my Jaeger

[centos@centos8 warmup-exercise]$ kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
grafana-7bdcf77687-w5hvt               1/1     Running   0          3h58m
istio-egressgateway-5547fcc8fc-qsd2l   1/1     Running   0          3h58m
istio-ingressgateway-8f568d595-j6wzd   1/1     Running   0          3h58m
istiod-6659979bdf-9chbn                1/1     Running   0          3h58m
jaeger-5c7c5c8d87-p5678                1/1     Running   0          3h58m
kiali-7fd9f6f484-vlxms                 1/1     Running   0          3h58m
prometheus-f5f544b59-br5n4             2/2     Running   0          3h58m

---

[centos@centos8 warmup-exercise]$ kubectl --namespace istio-system describe pod/jaeger-5c7c5c8d87-p5678
Name:         jaeger-5c7c5c8d87-p5678
Namespace:    istio-system
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Thu, 21 Oct 2021 12:42:47 -0400
Labels:       app=jaeger
              pod-template-hash=5c7c5c8d87
Annotations:  prometheus.io/port: 14269
              prometheus.io/scrape: true
              sidecar.istio.io/inject: false
Status:       Running
IP:           172.17.0.6
IPs:
  IP:           172.17.0.6
Controlled By:  ReplicaSet/jaeger-5c7c5c8d87
Containers:
  jaeger:
    Container ID:   docker://3e155e7909f9f9976184b0b8f72880307d6bb7f8810d98c25d2dd8f18df342bb
    Image:          docker.io/jaegertracing/all-in-one:1.20
    Image ID:       docker-pullable://jaegertracing/all-in-one@sha256:54c2ea315dab7215c51c1b06b111c666f594e90317584f84eabbc59aa5856b13
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 21 Oct 2021 12:49:26 -0400
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      10m
    Liveness:   http-get http://:14269/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:14269/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      BADGER_EPHEMERAL:            false
      SPAN_STORAGE_TYPE:           badger
      BADGER_DIRECTORY_VALUE:      /badger/data
      BADGER_DIRECTORY_KEY:        /badger/key
      COLLECTOR_ZIPKIN_HTTP_PORT:  9411
      MEMORY_MAX_TRACES:           50000
      QUERY_BASE_PATH:             /jaeger
    Mounts:
      /badger from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tj4pj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-tj4pj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason        Age                 From             Message
  ----     ------        ----                ----             -------
  Warning  Unhealthy     51m (x3 over 102m)  kubelet          Readiness probe failed: Get "http://172.17.0.6:14269/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy     51m (x5 over 160m)  kubelet          Liveness probe failed: Get "http://172.17.0.6:14269/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  NodeNotReady  51m                 node-controller  Node is not ready

---

[centos@centos8 warmup-exercise]$ kubectl get po -o wide

NAME                                  READY   STATUS    RESTARTS   AGE     IP            NODE       NOMINATED NODE   READINESS GATES

api-gateway-5cd5c547c6-lrt6k          2/2     Running   0          3h30m   172.17.0.13   minikube   <none>           <none>
photo-service-7c79458679-trblk        2/2     Running   0          3h30m   172.17.0.11   minikube   <none>           <none>
position-simulator-6c7b7949f8-k2z7t   2/2     Running   0          3h30m   172.17.0.14   minikube   <none>           <none>
position-tracker-cbbc8b7f6-dl4gz      2/2     Running   0          3h30m   172.17.0.12   minikube   <none>           <none>
staff-service-6597879677-7zh2c        2/2     Running   0          3h30m   172.17.0.15   minikube   <none>           <none>
vehicle-telemetry-c8fcb46c6-n9764     2/2     Running   0          3h30m   172.17.0.10   minikube   <none>           <none>
webapp-85fd946885-zdjck               2/2     Running   0          3h30m   172.17.0.16   minikube   <none>           <none>

---

I'm still learning devops. Let me know if I missed something.

Answer 1

The default sample rate for Jaeger is 1% : https://istio.io/latest/docs/tasks/observability/distributed-tracing/jaeger/