OTEL collector substituting JSON config as environment variable

Question

I'm looking to secure and automate the deployment of my otel-collector in my cluster by having it's sensitive configuration values injected by the vault-agent-injector mutating webhook. I'm using a customised otel-collector binary that ships with a google chronicle forwarder.

https://github.com/observIQ/bindplane-agent/tree/release/v1.49.0/exporter/chronicleexporter

Here is my config:

exporters:
  chronicle:
    creds: |
      ${env:CHRONICLE_JSON}
    customer_id: someCustomerId
    endpoint: someChronicleEndpoint
    log_type: CUSTOM_APPLICATION_ACCESS
  debug:
    verbosity: detailed
  elasticsearch:
    api_key: ${env:ES_API_KEY}
    endpoints:
    - someESEndpoint
    password: ${env:ES_PASSWORD}
    user: someESuser
  elasticsearch/log:
    endpoints:
    - someESEndpoint

When the collector substitutes the value of CHRONICLE_JSON, I get an unexpected end of input error from the collector. Here is the value of that env var with sensitive values replaced with placeholders.

{
  "type": "service_account",
  "project_id": "someProjectID",
  "private_key_id": "11111111111111111",
  "private_key": "-----BEGIN PRIVATE KEY-----\nsomeCert\n-----END PRIVATE KEY-----\n",
  "client_email": "someClientEmail",
  "client_id": "111111111111111111111",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "someCertUrl",
  "universe_domain": "googleapis.com"
}

I know this configuration works, as when I run the collector manually, the exporter forwards correctly. I know my Vault annotations are working, as in /vault/secrets/otel-collector, I can see my secrets being populated correctly, and I know my container is sourcing them correctly, as the environment variables are also populated.

I think I'm formatting something incorrectly here or not understanding exactly how this substitution is working. It seems to just be this JSON value that is causing issues as with only the JSON value hardcoded the collector also substitutes the ES values fine.

Here is the contents of the secret file that the injector templates, again with placeholders values:

export ES_API_KEY="someAPIKey"
export ES_PASSWORD="somePassword"
export CHRONICLE_JSON='{
  "type": "service_account",
  "project_id": "someProjectID",
  "private_key_id": "11111111111111111",
  "private_key": "-----BEGIN PRIVATE KEY-----\nsomeCert\n-----END PRIVATE KEY-----\n",
  "client_email": "someClientEmail",
  "client_id": "111111111111111111111",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "someCertUrl",
  "universe_domain": "googleapis.com"
}'

I have tried formatting the JSON in a single line and escaping it etc.. but I can't for the life of me get this to substitute correctly. I did read up on https://github.com/open-telemetry/opentelemetry-collector/issues/5223 but this is still an open issue so I can't see exactly what's going wrong with my configuration.

How do I format this env var so that it's valid for my collector?

Answer 1

I ended up injecting the json as a file and adjusting the exporter config to read from the file.

vault.hashicorp.com/agent-inject-secret-config.json: "secret/data/development/otel-collector"
vault.hashicorp.com/agent-inject-template-config.json: |
  {{`{{- with secret "secret/data/development/otel-collector" -}}`}}
  {{`{{ .Data.data.CHRONICLE_JSON }}`}}
  {{`{{- end }}`}}
vault.hashicorp.com/agent-inject-secret-envvars: "secret/data/development/otel-collector"
vault.hashicorp.com/agent-inject-template-envvars: |
  {{`{{- with secret "secret/data/development/otel-collector" -}}`}}
  export ES_PASSWORD="{{`{{ .Data.data.ES_PASSWORD }}`}}"
  export ES_API_KEY="{{`{{ .Data.data.ES_API_KEY }}`}}"
  {{`{{- end }}`}}

exporters:
  chronicle:
    creds_file_path: '/vault/secrets/config.json'