GCP API Gateway with an API Key fails with 403 error stating ... .cloud.goog is not enabled for the project

Question

First things first, let me show you some of my gcloud settings. When I run gcloud config list, this is my output:

[core]
account = <SERVICE ACCOUNT NAME>@<PROJECT NAME>.iam.gserviceaccount.com
disable_usage_reporting = True
project = <PROJECT NAME>

Your active configuration is: [default]

When I run gcloud services list, this is my output:

apigateway.googleapis.com            API Gateway API
artifactregistry.googleapis.com      Artifact Registry API
bigquery.googleapis.com              BigQuery API
bigquerymigration.googleapis.com     BigQuery Migration API
bigquerystorage.googleapis.com       BigQuery Storage API
cloudapis.googleapis.com             Google Cloud APIs
cloudbuild.googleapis.com            Cloud Build API
clouddebugger.googleapis.com         Cloud Debugger API
cloudfunctions.googleapis.com        Cloud Functions API
cloudresourcemanager.googleapis.com  Cloud Resource Manager API
cloudtrace.googleapis.com            Cloud Trace API
containerregistry.googleapis.com     Container Registry API
datastore.googleapis.com             Cloud Datastore API
eventarc.googleapis.com              Eventarc API
iam.googleapis.com                   Identity and Access Management (IAM) API
iamcredentials.googleapis.com        IAM Service Account Credentials API
logging.googleapis.com               Cloud Logging API
monitoring.googleapis.com            Cloud Monitoring API
oslogin.googleapis.com               Cloud OS Login API
pubsub.googleapis.com                Cloud Pub/Sub API
run.googleapis.com                   Cloud Run Admin API
secretmanager.googleapis.com         Secret Manager API
servicecontrol.googleapis.com        Service Control API
servicemanagement.googleapis.com     Service Management API
serviceusage.googleapis.com          Service Usage API
source.googleapis.com                Legacy Cloud Source Repositories API
sql-component.googleapis.com         Cloud SQL
storage-api.googleapis.com           Google Cloud Storage JSON API
storage-component.googleapis.com     Cloud Storage
storage.googleapis.com               Cloud Storage API
sts.googleapis.com                   Security Token Service API

I have an API Gateway with the following config file:

swagger: '2.0'
info:
  title: <API TITLE>
  description: API Gateway First for Sphrn Testing
  version: 1.0.0
securityDefinitions:
  api_key_header:
    type: apiKey
    name: x-api-key
    in: header
schemes:
  - https
produces:
  - application/json
paths:
  /entrypoint1:
    post:
      summary: Simple echo service
      operationId: <OPERATION ID HERE>
      x-google-backend:
        address: https://<CLOUD FUNCTION NAME>-<STRING I DON'T RECOGNIZE>-uc.a.run.app
      security:
        - api_key_header: []
      responses:
        '200':
          description: OK

I call the api from my command line with this script:

curl --location --request POST 'https://<API CALLABLE ENDPOINT>.uc.gateway.dev/endpoint1' \
--header 'X-goog-api-key: <MY API KEY HERE>' \
--header 'Content-Type: application/json; charset=utf-8' \
--data-raw '{
    "name": "Test1"
}'

but it fails with this in my terminal:

{"code":403,"message":"PERMISSION_DENIED:API <SERVICE ACCOUNT NAME>-<STRING I DON'T RECOGNIZE>.apigateway.<PROJECT NAME>.cloud.goog is not enabled for the project."}

My API key looks like this:

And I went into the logs explorer for the API Gateway endpoint and this is the more detailed logs from my 403 failed curl command (sanitized for identifying information of course):

{
    "httpRequest": {
        "latency": "0.040s",
        "protocol": "http",
        "remoteIp": "<MY IP ADDRESS>",
        "requestMethod": "POST",
        "requestSize": "1053",
        "requestUrl": "/endpoint1",
        "responseSize": "346",
        "status": 403
    },
    "insertId": "<LONG GUID LOOKING STRING>@a1",
    "jsonPayload": {
        "api_key": "<MY API KEY>",
        "api_key_state": "NOT ENABLED",
        "api_method": "1.<API ID>_<STRING I DON'T RECOGNIZE>_apigateway_<PROJECT NAME>_cloud_goog.<OPERATIONID FROM CONFIG YAML>",
        "api_name": "1.<API ID>_<STRING I DON'T RECOGNIZE>_apigateway_<PROJECT NAME>_cloud_goog",
        "api_version": "1.0.0",
        "error_cause": "API <API ID>_<STRING I DON'T RECOGNIZE>.apigateway.<PROJECT NAME>.cloud.goog is not enabled for the project.",
        "http_status_code": 403,
        "location": "us-central1",
        "log_message": "1.<API ID>_<STRING 1 I DON'T RECOGNIZE>_apigateway_<PROJECT NAME>_cloud_goog.<OPERATIONID FROM CONFIG YAML> is called",
        "producer_project_id": "<PROJECT NAME>",
        "response_code_detail": "service_control_check_error{SERVICE_NOT_ACTIVATED}",
        "service_agent": "ESPv2/2.40.0",
        "service_config_id": "<CONFIGURATION ID>",
        "timestamp": "<TIMESTAMP HERE AS DECIMAL>"
    },
    "logName": "projects/<PROJECT NAME>/logs/<API ID>_<STRING I DON'T RECOGNIZE>.apigateway.<PROJECT NAME>.cloud.goog%2Fendpoints_log",
    "receiveTimestamp": "<TIMESTAMP HERE AS STRING>",
    "resource": {
        "labels": {
            "location": "us-central1",
            "method": "1.<API ID>-<STRING I DON'T RECOGNIZE>_apigateway_<PROJECT NAME>_cloud_goog.<OPERATIONID FROM CONFIG YAML>",
            "project_id": "<PROJECT NAME>",
            "service": "<API ID>-<STRING I DON'T RECOGNIZE>.apigateway.<PROJECT NAME>.cloud.goog",
            "version": "1.0.0"
        },
        "type": "api"
    },
    "severity": "ERROR",
    "timestamp": "<TIMESTAMP HERE AS STRING>"
}

So how do I get this curl to succeed...? I'm assuming it's a permissions issue, but what permission does my service account not have?

When I run:

gcloud projects get-iam-policy <PROJECT ID> \
--flatten="bindings[].members" \
--format='table(bindings.role)' \
--filter="bindings.members:<SERVICE ACCOUNT NAME>@<PROJECT NAME>.iam.gserviceaccount.com"

I get this output:

ROLE
roles/cloudfunctions.serviceAgent
roles/serviceusage.serviceUsageViewer

Answer 1

In addition to Christian's answer, the terraform solution for it would be:

modules/api_gateway/main.tf

resource "google_api_gateway_api" "api_gw" {
    provider = google-beta
    api_id = var.api_id
}

# In addition to API Gateway Gateway and API Config

resource "google_project_service" "express" {
  depends_on = [ google_api_gateway_gateway.api_gw ]
  service = google_api_gateway_api.api_gw.managed_service

  timeouts {
    create = "30m"
    update = "40m"
  }

  disable_dependent_services = true
}

modules/api_gateway/outputs.tf

output "api-managed-service" {
    value = google_api_gateway_api.api_gw.managed_service
}

modules/api_key/main.tf

resource "google_apikeys_key" "api_key" {
    name         = var.api_key_name
    display_name = var.api_key_display_name

    restrictions {
        api_targets {
            service = "apigateway.googleapis.com" 
            }
        api_targets {
            service = var.api_target_service
            methods = ["GET*"]
            }
        api_targets {
            service = "cloudfunctions.googleapis.com" 
            }
 }
}

main.tf

module "api-key" {
    source = "./modules/api_key"
    api_key_name = var.api_key_name
    api_key_display_name = var.api_key_display_name
    api_target_service = module.api-gateway.api-managed-service
}

Answer 2

I had to enable the service by using my actual "master" Gmail account with which I created the GCP project and enabling the service <SERVICE ACCOUNT NAME>-....apigateway.<PROJECT NAME>.cloud.goog via gcloud commands. Then I had 1 more problem where I didn't enable the operationId listed in my openapi config yaml file in the API key restrictions menu.

I'm assuming anyone reading this has already logged in with their service account via gcloud auth login and activated their relevant service account with gcloud auth activate-service-account <SERVICE ACCOUNT NAME>@<PROJECT NAME>.iam.gserviceaccount.com --key-file=/path/to/keyfile.json

Enable Service Fix
I switched my gcloud account to my "master" account with gcloud config set account <MASTER GCLOUD ACCOUNT NAME>@gmail.com, then:

gcloud services enable <SERVICE ACCOUNT NAME>-....apigateway.<PROJECT NAME>.cloud.goog \
--project=<PROJECT ID (THE NUMBER NOT THE TEXT NAME>

This made it so calling the API with my API key in the header give me a new error {"message":"PERMISSION_DENIED: The API targeted by this request is invalid for the given API key.","code":403}

operationId API Restriction Menu Fix
I had to enable the operationId listed in my openapi config yaml file in the API key restrictions menu. After that it appeared in the "Selected APIs" section of the API Key Credentials page:

After making this change, my curl request:

curl --location --request POST 'https://<API CALLABLE ENDPOINT>.uc.gateway.dev/endpoint1' \
--header 'X-goog-api-key: <MY API KEY HERE>' \
--header 'Content-Type: application/json; charset=utf-8' \
--data-raw '{
    "name": "Test1"
}'

worked perfectly!