GKE tries to reserve static IP that already exists which results in ERRORS

Question

We are encountering a constant errors about creation of the static IP address.

What we've done is:

1. With the usage of Terraform we have reserved a static IP address.
2. Then we have deployed the emissary ingress onto our GKE cluster.
3. As instructed in the dcocs we've set the spec.loadBalancerIP to the IP reserved by Terraform.

After that, every 10 minutes, we noticed that GKE is trying to reserve the exact same IP address that we defined in the service and fails with the following error:

Invalid value for field 'resource.address': '<IP-ADDRESS>'. Specified IP address is already reserved.

When I deleted the service the errors stopped.

Currently we are running version 1.29.9-gke.1496000, but the first time we noticed the error was in ~June and from that time, we've upgraded version of the control plane, so I'm not sure the version is an issue in that case.

I am attaching the full log from the GCP Logs Explorer in the JSON format with all the sensitive values redacted.

Besides that, everything is working as expected. Forwarding rules and backends are created correctly, traffic is passing through - all good. The only issue is that we are getting ERROR entries in the logs.

{
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "status": {
        "code": 3,
        "message": "Invalid value for field 'resource.address': '<RESERVED-IP>'. Specified IP address is already reserved."
      },
      "authenticationInfo": {
        "principalEmail": "service-<PROJECT-NUMBER>@container-engine-robot.iam.gserviceaccount.com",
        "principalSubject": "serviceAccount:service-<PROJECT-NUMBER>@container-engine-robot.iam.gserviceaccount.com"
      },
      "requestMetadata": {
        "callerIp": "34.91.169.220",
        "callerSuppliedUserAgent": "google-api-go-client/0.5 Kubernetes/0.0.0 (linux amd64),gzip(gfe)",
        "requestAttributes": {
          "time": "2024-11-30T13:54:57.570604Z",
          "auth": {}
        },
        "destinationAttributes": {}
      },
      "serviceName": "compute.googleapis.com",
      "methodName": "v1.compute.addresses.insert",
      "authorizationInfo": [
        {
          "resource": "projects/<PROJECT-ID>/regions/europe-west4/addresses/<ADDRESS-ID>",
          "permission": "compute.addresses.create",
          "granted": true,
          "resourceAttributes": {
            "service": "compute",
            "name": "projects/<PROJECT-ID>/regions/europe-west4/addresses/<ADDRESS-ID>",
            "type": "compute.addresses"
          },
          "permissionType": "ADMIN_WRITE"
        }
      ],
      "resourceName": "projects/<PROJECT-ID>/regions/europe-west4/addresses/<ADDRESS-ID>",
      "request": {
        "addressType": "EXTERNAL",
        "@type": "type.googleapis.com/compute.addresses.insert",
        "description": "{\"kubernetes.io/service-name\":\"emissary/emissary-ingress\"}",
        "address": "<RESERVED-IP>",
        "name": "<ADDRESS-ID>",
        "networkTier": "PREMIUM"
      },
      "response": {
        "error": {
          "errors": [
            {
              "domain": "global",
              "reason": "invalid",
              "message": "Invalid value for field 'resource.address': '<RESERVED-IP>'. Specified IP address is already reserved."
            }
          ],
          "code": 400,
          "message": "Invalid value for field 'resource.address': '<RESERVED-IP>'. Specified IP address is already reserved."
        },
        "@type": "type.googleapis.com/error"
      },
      "resourceLocation": {
        "currentLocations": [
          "europe-west4"
        ]
      }
    },
    "insertId": "-tmww84ef5tuc",
    "resource": {
      "type": "gce_reserved_address",
      "labels": {
        "location": "europe-west4",
        "reserved_address_id": "",
        "project_id": "<PROJECT-ID>"
      }
    },
    "timestamp": "2024-11-30T13:54:57.331731Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/root_trigger_id": "4be0d0ce-ad68-4b10-9480-a0d9c2648ef5"
    },
    "logName": "projects/<PROJECT-ID>/logs/cloudaudit.googleapis.com%2Factivity",
    "receiveTimestamp": "2024-11-30T13:54:57.602602513Z"
  }

Also posted at https://issuetracker.google.com/issues/381518936