CODEWITHSUNDEEP
terraform
thclpr
thclpr

Reputation: 5938

Unable to pass service annotations when deploying helm chart via terraform

I have seeing some examples regarding how to pass annotations when deploying a helm chart via terraform but none of then are working as expected, in this case, im trying to create a service assining a private ip on a specific subnet, but instead, its creating a public IP. My terraform files:

locals {
  helm_general = {
    # Reference values
    # https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
    elasticsearch = {
      name      = "elasticsearch"
      chart     = "elastic/elasticsearch"
      tag       = "7.14.0"
      namespace = "elasticsearch"
      set = [
        {
          name  = "nodeSelector.agentpool"
          value = "general"
        },      
        {
          name  = "replicas"
          value = "1"
        },
        {
          name  = "minimumMasterNodes"
          value = "1"
        },
        {
          name  = "image"
          value = "docker.elastic.co/elasticsearch/elasticsearch"
        },
        {
          name  = "imageTag"
          value = "7.14.0"
        },
        {
          name  = "resources.requests.cpu"
          value = "10m"
        },
        {
          name  = "resources.requests.memory"
          value = "128Mi"
        },
        {
          name  = "volumeClaimTemplate.reosources.requests.storage"
          value = "4Gi"
        },
        {
          name  = "persistence.enabled"
          value = "false"
        },
        {
          name  = "service.type"
          value = "LoadBalancer"
        },
        {
          name  = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
          value = "true"
        },
        {
          name  = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal-subnet"
          value = "somesubnet"
        },          
      ]
      timeout = "900"
    }
  }
}

Helm deployment

resource "helm_release" "helm" {
  provider  = helm.general
  for_each  = local.helm_general
  name      = each.value.name
  chart     = each.value.chart
  namespace = format(each.value.namespace)
  dynamic "set" {
    iterator = item
    for_each = each.value.set == null ? [] : each.value.set

    content {
      name  = item.value.name
      value = item.value.value
    }
  }
  depends_on = [kubernetes_namespace.general]
}

Plan / apply output https://i.sstatic.net/NneuP.png

And what is currently being deployed is a public ip instead of a private ip:

Namespace:                elasticsearch                                                     
Labels:                   app=elasticsearch-master                                          
                          app.kubernetes.io/managed-by=Helm                                 
                          chart=elasticsearch                                               
                          heritage=Helm                                                     
                          release=elasticsearch                                             
Annotations:              meta.helm.sh/release-name: elasticsearch                          
                          meta.helm.sh/release-namespace: elasticsearch                     
Selector:                 app=elasticsearch-master,chart=elasticsearch,release=elasticsearch
Type:                     LoadBalancer                                                      
IP Families:              <none>                                                            
IP:                       xx                                                      
IPs:                      xxx                                                     
LoadBalancer Ingress:     redacted public ip                                                      
Port:                     http  9200/TCP                                                    
TargetPort:               9200/TCP                                                          
NodePort:                 http  32083/TCP                                                   
Endpoints:                                                                                  
Port:                     transport  9300/TCP                                               
TargetPort:               9300/TCP                                                          
NodePort:                 transport  32638/TCP                                              
Endpoints:                                                                                  
Session Affinity:         None                                                              
External Traffic Policy:  Cluster                                                           
Events:                                                                                     
  Type    Reason                Age        From                Message                      
  ----    ------                ----       ----                -------                      
  Normal  EnsuringLoadBalancer  1s         service-controller  Ensuring load balancer       
  Normal  EnsuredLoadBalancer   <invalid>  service-controller  Ensured load balancer

References that i have been following:

https://github.com/hashicorp/terraform-provider-helm/issues/125 https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release

Edit:

error message:

Error: unable to decode "": resource.metadataOnlyObject.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects " or n, but found t, error found in #10 byte of ...|nternal":true},"labe|..., bigger context
..|beta.kubernetes.io/azure-load-balancer-internal":true},"labels":{"app":"elasticsearch-master","chart|...

  with helm_release.helm["elasticsearch"],
  on aks-general-helm.tf line 1, in resource "helm_release" "helm":
   1: resource "helm_release" "helm" {

Upvotes: 7

Views: 7772

Answers (5)

Anderson Bispo
Anderson Bispo

Reputation: 21

Another way of doing this without using escape:

resource "helm_release" "coroot" {
  name       = "coroot"

  namespace =  "coroot"
  create_namespace = true
  repository = "https://coroot.github.io/helm-charts"
  chart      = "coroot"

  set {
    name = "corootCE.service.type"
    value = "LoadBalancer"
  }

  values = [
        <<-EOT
          corootCE:
            service:
              annotations:
                oci.oraclecloud.com/load-balancer-type: "lb"
                service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
                service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "10"
                service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "10"
      EOT
  ]

}

the result:

Annotations:              meta.helm.sh/release-name: coroot
                          meta.helm.sh/release-namespace: coroot
                          oci.oraclecloud.com/load-balancer-type: lb
                          service.beta.kubernetes.io/oci-load-balancer-internal: true
                          service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                          service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: 10
                          service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: 10

Upvotes: 2

antonioua
antonioua

Reputation: 11

Here is the way how I add annotations to ArgoCD k8s service:

resource "helm_release" "argocd" {
...
  dynamic "set" {
    for_each = local.argocd_service_annotations
    content {
      // server.service.annotations
      name = "argo-cd.server.service.annotations.${replace(set.key, ".", "\\.")}"
      value = set.value
    }
  }
}

  argocd_service_annotations = {
    "external-dns.alpha.kubernetes.io/hostname" : "myhost.com"
  }

Result:

argo-cd:
  server:
    service:
      annotations:
        external-dns.alpha.kubernetes.io/hostname: myhost.com

Upvotes: 0

ishuar
ishuar

Reputation: 1328

One way of doing this without the escape characters and keeping the original YAML format would be using values attribute of the helm_release resource. Would be curious to know if there was a specific used-case to not use it in the first place.

resource "helm_release" "helm" {
  provider  = helm.general
  for_each  = local.helm_general
  name      = each.value.name
  chart     = each.value.chart
  namespace = format(each.value.namespace)

  values = each.value.values ##CHANGE IS HERE ##

  dynamic "set" {
    iterator = item
    for_each = each.value.set == null ? [] : each.value.set

    content {
      name  = item.value.name
      value = item.value.value
    }
  }
  depends_on = [kubernetes_namespace.general]
}

The local in your case would be adjusted to below, you can still keep set for something which has a dependency on any terraform resources or with any other logical reasons.

locals {
  helm_general = {
    # Reference values
    # https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
    elasticsearch = {
      [...]
      values = [file("${path.module}/elasticsearch-values.yaml")]
      [...]
    }
  }
}

There has to be a new file elasticsearch-values.yaml at the same path (which can be adjusted with any relative path as per the local.helm_general.elasticsearch.values) location, where this terraform configurations exist.

# Reference values can be adapted as per the upstream chart.
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
service:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "somesubnet"

Upvotes: 0

somesh koli
somesh koli

Reputation: 11

if you have multiple you can try something like this, pass annotations as map and set them using

  dynamic "set" {
    for_each = var.ingress_annotations
    content {
      name = replace(set.key, ".", "\\.")
      value = set.value
    }
  }

Upvotes: -1

Ricardo Alvial
Ricardo Alvial

Reputation: 106

I just faced a similar issue, and here is what worked for me:

{
 name  = "service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
 value = "true"
},

I think the issue is how it is concatenated. The service in the chart manifest for elastic official is service.annotations:{} so you need to append .service then use \\.

Upvotes: 9

Related Questions