Getting 'invalid bearer token' error when integrating webhook-token with Kube-apiserver: How to resolve?

Question

I have created a go server to return tokenreview JSON serialized string as per the docs to validate users using id and password. My go server is able to send back the correct response but it seems like there is some issue with validation which I am not able to understand. Particulary, I have no idea what the above error means..

kube-apiserver logs

I0530 12:17:07.582993       1 request.go:1188] Request Body: {"kind":"TokenReview","apiVersion":"authentication.k8
s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"token":"test:test1234","audiences":["https://kubernet
es.default.svc.cluster.local"]},"status":{"user":{}}}
I0530 12:17:07.583057       1 round_trippers.go:463] POST https://webhook-token.nginx.svc.cluster.local:8080/?timeout=30s
I0530 12:17:07.583067       1 round_trippers.go:469] Request Headers:
I0530 12:17:07.583081       1 round_trippers.go:473]     Accept: application/json, */*
I0530 12:17:07.583091       1 round_trippers.go:473]     Content-Type: application/json
I0530 12:17:07.597571       1 round_trippers.go:574] Response Status: 200 OK in 14 milliseconds
I0530 12:17:07.597769       1 round_trippers.go:577] Response Headers:
I0530 12:17:07.597793       1 round_trippers.go:580]     Content-Type: Application/json
I0530 12:17:07.597840       1 round_trippers.go:580]     Content-Length: 199
I0530 12:17:07.597855       1 round_trippers.go:580]     Date: Tue, 30 May 2023 12:17:07 GMT
I0530 12:17:07.597941       1 request.go:1188] Response Body: {"apiVersion":"authentication.k8s.io/v1beta1","kind":"TokenReview","Status":{"authenticated":true,"user":{"username":"test","uid":"100"},"audiences":["https://kubernetes.default.svc.cluster.local"]}}
E0530 12:17:07.598332       1 authentication.go:70] "Unable to authenticate the request" err="invalid bearer token"

auth-config.yaml

apiVersion: v1
kind: Config
clusters:
  - name: authn
    cluster:
      server: https://webhook-token.nginx.svc.cluster.local:8080
      certificate-authority: /etc/pki/ca.pem
users:
  - name: kube-apiserver
    user:
      client-certificate: /etc/pki/client.pem # cert for the webhook plugin to use
      client-key: /etc/pki/client.key.pem          # key matching the cert
contexts:
- context:
    cluster: authn
    user: kube-apiserver
  name: authn
current-context: authn

/etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.128.0.3:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --v=8
    - --authentication-token-webhook-version=v1beta1
    - --authentication-token-webhook-config-file=/etc/auth-config.yaml
    - --advertise-address=10.128.0.3
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.27.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 10.128.0.3
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 10.128.0.3
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 10.128.0.3
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
    - mountPath: /etc/auth-config.yaml
      name: webhook-token
      readOnly: true
    - mountPath: /etc/pki/ca.pem
      name: webhook-ca
      readOnly: true
    - mountPath: /etc/pki/client.pem
      name: client-cert
      readOnly: true
    - mountPath: /etc/pki/client.key.pem
      name: clientkey-cert
      readonly: true
  hostNetwork: true
  priority: 2000001000
  hostAliases:
  - ip: 10.98.16.119
    hostnames:
    - "webhook-token.nginx.svc.cluster.local"
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
  - hostPath:
      path: /etc/auth-config.yaml
    name: webhook-token
  - hostPath:
      path: /etc/pki/ca.pem
    name: webhook-ca
  - hostPath:
      path: /etc/pki/client.pem
    name: client-cert
  - hostPath:
      path: /etc/pki/client.key.pem
    name: clientkey-cert
status: {}

Answer 1

It seems I needed to encode the token in base64 format before sending request to the kube-apiserver.