HPA unable to read metrics from Google Managed Prometheus

Question

We were trying to use metrics exported from our app to Google Manager Prometheus. Even though we are able to view the metrics in Google Monitoring, the HPA is not able to read the values and we are getting an error: "HPA cannot read metric value"

Attaching screenshots of kubectl describe hpa -n aries and Google Monitoring

Tried configuring the HPA using the GCP console but still got the save error

Edit 1

GKE Version: 1.27.14-gke.1059002

As of now I am able to get CPU resource based HPA to work.

This is the YAML file I am using to manage the HPA settings

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: aries-celery-hpa
  namespace: aries
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: aries-celery
  minReplicas: {{ .Values.spec_celery.hpa.min_replicas }}
  maxReplicas: {{ .Values.spec_celery.hpa.max_replicas }}
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: {{ .Values.spec_celery.hpa.metrics.cpu.averageUtilization}}

  - type: External
    external:
      metric:
        name: prometheus.googleapis.com|aries_backlog_pressure|gauge
      target:
        type: AverageValue
        averageValue: 2

metrics.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: {{ .Values.app_name }}-metric-exporter
  name: {{ .Values.app_name }}-metric-exporter
  namespace: {{ .Values.namespace }}
spec:
  replicas: {{ .Values.spec.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.app_name }}-metric-exporter
  template:
    metadata:
      labels:
        app: {{ .Values.app_name }}-metric-exporter
    spec:
      tolerations:
      - key: "low-cpu"
        operator: "Equal"
        effect: "NoSchedule"
        value: "true"

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "low-cpu"
                operator: "In"
                values:
                - "true"
      containers:
        - image: {{ .Values.image.name }}:{{ .Values.image.tag }}
          imagePullPolicy: IfNotPresent
          command: ["python3", "metrics.py"]
          name: {{ .Values.app_name }}-metric-exporter
          ports:
            - name: metrics
              containerPort: 8080
          volumeMounts:
            - name: aries-config
              readOnly: true
              mountPath: "/app/.env"
              subPath: .env
          resources:
            limits:
              cpu: {{ .Values.spec.resources.limits.cpu }}
              memory: {{ .Values.spec.resources.limits.memory }}

            requests:
              cpu: {{ .Values.spec.resources.requests.cpu }}
              memory: {{ .Values.spec.resources.requests.memory }}

      terminationGracePeriodSeconds: 10
      volumes:
        - name: aries-config
          configMap:
            name: aries-config
            items:
              - key: .env
                path: .env
            defaultMode: 420


---

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.app_name }}-metric-exporter
  namespace: aries
spec:
  selector:
    app: {{ .Values.app_name }}-metric-exporter
  ports:
  - protocol: TCP
    port: 80
    targetPort: metrics
  type: LoadBalancer

---

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: {{ .Values.app_name }}-metric-exporter
  namespace: {{ .Values.namespace }}
  labels:
    app.kubernetes.io/name: {{ .Values.app_name }}-metric-exporter
spec:
  selector:
    matchLabels:
      app: {{ .Values.app_name }}-metric-exporter
  endpoints:
  - port: metrics
    interval: 30s