Difference between discoveredLabels and labels in Prometheus targets endpoint

Question

I have a question concerning the labeling of Prometheus. When I call the targets endpoint (<prometheus address>/api/v1/targets), I get a json containing the scrape targets including different key-value pairs. What is the difference between the keys "discoveredLabels" and "labels" in a given context. When I query a metric which comes from a specific target, Prometheus seems to include the labels listed in the "labels" key. Where are the "discoveredLabels used"? A sample extract (not complete) which includes both keys is shown below.

{u'discoveredLabels': {u'__address__': u'12.13.0.3:9090',
                                                     u'__meta_kubernetes_namespace': u'monitoring',
                                                     u'__meta_kubernetes_pod_annotation_prometheus_io_port': u'9090',
                                                     u'__meta_kubernetes_pod_annotation_prometheus_io_scrape': u'true',
                                                     u'__meta_kubernetes_pod_container_name': u'prometheus',
                                                     u'__meta_kubernetes_pod_container_port_name': u'',
                                                     u'__meta_kubernetes_pod_container_port_number': u'9090',
                                                     u'__meta_kubernetes_pod_container_port_protocol': u'TCP',
                                                     u'__meta_kubernetes_pod_host_ip': u'11.0.3.12',
                                                     u'__meta_kubernetes_pod_ip': u'132.14.0.1',
                                                     u'__meta_kubernetes_pod_label_name': u'prometheus',
                                                     u'__meta_kubernetes_pod_label_pod_template_hash': u'6bwedd76d9',
                                                     u'__meta_kubernetes_pod_name': u'prometheus-6bwedd76d9-nxlvr',
                                                     u'__meta_kubernetes_pod_node_name': u'minikube',
                                                     u'__meta_kubernetes_pod_ready': u'true',
                                                     u'__metrics_path__': u'/metrics',
                                                     u'__scheme__': u'http',
                                                     u'job': u'kubernetes-pods'},
                               u'health': u'up',
                               u'labels': {u'instance': u'12.13.0.3:9090',
                                           u'job': u'kubernetes-pods',
                                           u'kubernetes_namespace': u'monitoring',
                                           u'kubernetes_pod_name': u'prometheus-6bwedd76d9-nxlvr',
                                           u'name': u'prometheus',
                                           u'pod_template_hash': u'6bwedd76d9'},

Answer 1

discoveredLabels is everything that comes from service discovery which you'll have available to you in target relabelling. labels is the output of that process, which are the ultimate target labels.