Vault Agent Template doesn't render with message missing dependency: vault.write

Question

Vault agent v1.4.2 service not able to render template and create files when running under the systemctl.

But the same template rendering works as expected with command used in vault agent service

/opt/vault/bin/vault agent -config /opt/vault/config/default.hcl -log-level=info

So it looks OS systemctl is not able to execute the command in the way when it get executed directly.

cat /etc/systemd/system/vault.service
[Unit]
Description=\"HashiCorp Vault Agent\"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/opt/vault/config/default.hcl
[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/opt/vault/bin/vault agent -config /opt/vault/config/default.hcl -log-level=debug
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

journalctl -u vault
  [DEBUG] (runner) running initial templates
 [DEBUG] (runner) initiating run
 [DEBUG] (runner) checking template 749c1d765e84e3e67f9dbb98ec983bf1
 [DEBUG] (runner) missing data for 1 dependencies
 **[DEBUG] (runner) missing dependency: vault.write(pki_int/test/issue/com -> 02a1cc85)**
 [DEBUG] (runner) add used dependency vault.write(pki_int/test/issue/com -> 02a1cc85) to missing since isLeader but do not have a watcher
 [DEBUG] (runner) was not watching 1 dependencies
 [DEBUG] (watcher) adding vault.write(pki_int/test/issue/com -> 02a1cc85)
 [DEBUG] (runner) checking template 43304cf2b8e3710476a7972b03a7544e
 [DEBUG] (runner) missing data for 1 dependencies
 [DEBUG] (runner) missing dependency: vault.write(pki_int/test/issue/com -> 02a1cc85)
 [DEBUG] (runner) missing data for 1 dependencies
 [DEBUG] (runner) checking template ca2b67db58c83f0e184663098bcb74b8
 [DEBUG] (runner) rendering "(dynamic)" => "/tmp/abc.test"
 [INFO] (runner) rendered "(dynamic)" => "/tmp/abc.test"
 [DEBUG] (runner) diffing and updating dependencies
 [DEBUG] (runner) watching 1 dependencies
 [INFO]  auth.handler: renewed auth token
 [DEBUG] Found certificate and set lease duration to 150 seconds
 [DEBUG] (runner) receiving dependency vault.write(pki_int/test/issue/com -> 02a1cc85)
 [DEBUG] (runner) initiating run
 [DEBUG] (runner) checking template 749c1d765e84e3e67f9dbb98ec983bf1
 [DEBUG] (runner) rendering "/opt/vault/templates/test.cert.tpl" => "/tmp/test.cert.pem"
 [INFO] (runner) rendered "/opt/vault/templates/test.cert.tpl" => "/tmp/test.cert.pem"
 [DEBUG] (runner) checking template 43304cf2b8e3710476a7972b03a7544e
 [DEBUG] (runner) rendering "/opt/vault/templates/test.key.tpl" => "/tmp/test.key.pem"
 [INFO] (runner) rendered "/opt/vault/templates/test.key.tpl" => "/tmp/test.key.pem"
 [DEBUG] (runner) checking template ca2b67db58c83f0e184663098bcb74b8
 [DEBUG] (runner) rendering "(dynamic)" => "/tmp/abc.test"
 [DEBUG] (runner) diffing and updating dependencies
 [DEBUG] (runner) vault.write(pki_int/test/issue/com -> 02a1cc85) is still needed
 [DEBUG] (runner) watching 1 dependencies
 [DEBUG] (runner) all templates rendered

Templates

test-cert.tpl
{{- /* test.abc.com.cert.tpl */ -}}
{{ with secret "pki_int/test/issue/abc.com" "common_name=test.abc.com" "ttl=2m" }}
{{ .Data.certificate }}
{{ .Data.issuing_ca }}{{ end }}

test-key.tpl
{{ with secret "pki_int/test/issue/abc.com" "common_name=test.abc.com" "ttl=2m" }}
{{ .Data.private_key }}{{ end }}
{{- /* test.abc.com.key.tpl */ -}}

test.tpl
{{ with secret "pki_int/test/issue/abc.com" "common_name=test.abc.com" "ttl=2m" }}
{{ .Data.private_key }}{{ end }}
abc123
To Reproduce
Steps to reproduce the behavior:

Run systemctl restart vault
Run journalctl -u vault
See error
Expected behavior
I expected it to create the following files with certs

/tmp/test.cert.pem
/tmp/test.key.pem
/tmp/abc.test"

Environment:

Vault Server Version (retrieve with vault status): 1.4.2

Vault CLI Version (retrieve with vault version): v1.4.2

Server Operating System/Architecture: Red Hat Enterprise Linux Server release 7.8 (Maipo)]

Vault agent configuration file(s):

cat /opt/vault/config/default.hcl
pid_file   = "/opt/vault/data/vault-pid"
vault {
  address = "https://xxxxxxxxx:443"
}
auto_auth {
  method "aws" {
    mount_path = "auth/aws"
    config = {
      type = "iam"
      role = "test-iam-role"
    }
  }
  sink "file" {
    config = {
      path = "/opt/vault/data/vault-token"
    }
  }
}
template {
  source      = "/opt/vault/templates/test.cert.tpl"
  destination = "/tmp/test.cert.pem"
  perms       = "0600"
}

template {
  source      = "/opt/vault/templates/test.key.tpl"
  destination = "/tmp/test.key.pem"
  perms       = "0600"
}

template {
  contents     = "testabc"
  destination = "/tmp/abc.test"
}

Additional context

I have mask the data to remove reference of original domain so please consider it if there is any mismatch.

Answer 1

The systemd config you shared has PrivateTmp=yes set, this will make files written to tmp visible only to Vault and not to other processes (see systemd docs). Removing this value will allow you to write to tmp and have other processes read the output.

As a side note, that looks like a systemd config designed for Vault server rather than the Vault agent - which is likely why it is so locked down.

Answer 2

I manage to solve this problem by changing the location of the certificate files from '/tmp' to '/opt/vault/certs' and it sorted the issue.