CODEWITHSUNDEEP
kubernetes
RubenLaguna
RubenLaguna

Reputation: 24696

Is there any mechanism in kubernetes to automatically add annotation to new pods in a specific namespace?

I have a namespace where new short-lived pods (< 1 minute) are created constantly by Apache Airflow. I want that all those new pods are annotated with aws.amazon.com/cloudwatch-agent-ignore: true automatically so that no CloudWatch metrics (container insights) are created for those pods.

I know that I can achieve that from airflow side with pod mutation hook but for the sake of the argument let's say that I have no control over the configuration of that airflow instance.

I have seen MutatingAdmissionWebhook and it seem that could do the trick, but it seems that it's considerable effort to set up. So I'm looking for a more of the shelf solution, I want to know if there is some "standard" admission controller that can do this specific use case, without me having to deploy a web server and implement the api required by MutatingAdmissionWebhook.

Is there any way to add that annotation from kubernetes side at pod creation time? The annotation must be there "from the beginning", not added 5 seconds later, otherwise the cwagent might pick it between the pod creation and the annotation being added.

Upvotes: 4

Views: 3949

Answers (2)

kkopczak
kkopczak

Reputation: 862

To clarify I am posting community Wiki answer.

You had to use aws.amazon.com/cloudwatch-agent-ignore: true annotation. This means the pod that has one, it will be ignored by amazon-cloudwatch-agent / cwagent.

Here is the excerpt of your solution how to add this annotation to Apache Airflow:

(...) In order to force Apache Airflow to add the aws.amazon.com/cloudwatch-agent-ignore: true annotation to the task/worker pods and to the pods created by the KubernetesPodOperator you will need to add the following to your helm values.yaml (assuming that you are using the "official" helm chart for airflow 2.2.3):

airflowPodAnnotations:
  aws.amazon.com/cloudwatch-agent-ignore: "true"
airflowLocalSettings: |-
  def pod_mutation_hook(pod): 
    pod.metadata.annotations["aws.amazon.com/cloudwatch-agent-ignore"] = "true"

If you are not using the helm chart then you will need to change the pod_template_file yourself to add the annotation and you will also need to modify the airflow_local_settings.py to include the pod_mutation_hook.

Here is the link to your whole answer.

Upvotes: 1

gohm&#39;c
gohm&#39;c

Reputation: 15490

You can try this repo which is a mutating admission webhook that does this. To date there's no built-in k8s support to do automatic annotation for specific namespace.

Upvotes: 1

Related Questions