GPG agent asks for passphrase when used in Docker container

Question

How can I stop GPG agent from always asking for a passphrase when run in a container?

I've been trying to resolve this for months (a year now?) On my host machine, I can ssh to github without issue - it uses my key, no questions asked (maybe the first time after boot it asks.) When I run git through an app in a docker container though, I get a prompt for my passphrase.

e.g. This will prompt me for my passphrase:

❯ dkcr --rm -it my_container cmd_that_will_run_many_git_remote_commands

BUT - and why this has been so difficult to debug - if I simply invoke SSH directly, I'm not prompted:

❯ dkcr --rm -it my_container ssh git@github.com
Warning: Permanently added the ECDSA host key for IP address '140.82.113.4' to the list of known hosts.
PTY allocation request failed on channel 0
Hi matt! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

Same is true if I just run bash, i.e. I can manually ssh/clone from github, but when I run commands that'll invoke git with ssh, I'm prompted for a passphrase when that app invokes git ls-remote ssh://git@github.com/..... If I enter the passphrase, it seems to work until I invoke the command again.

I share my agent with the container via the compose setup:

environment:
 - SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock
volumes:
 - ${SSH_AUTH_SOCK}:/run/host-services/ssh-auth.sock

I've even tried sharing the sshcontrol and gpg-agent.conf files even though this wouldn't make much sense to do - to no avail of course.

I don't share GPG_TTY nor GPG_AGENT_INFO, I'm not sure these would make sense to share

I've added debug logging to my gpg-agent, but nothing in the log has ever stood out to me (I don't know how to interpret the messages it outputs)

log-file /tmp/gpg-agent.log
debug-level advanced

This seems suspicious though:

2024-10-11 10:55:06 gpg-agent[8602] failed to build S-Exp (off=1436): Cannot allocate memory
2024-10-11 10:55:06 gpg-agent[8602] failed to read the secret key
2024-10-11 10:55:06 gpg-agent[8602] ssh sign request failed: Cannot allocate memory <gcrypt>
2024-10-11 10:55:06 gpg-agent[8602] ssh request handler for sign_request (13) ready
2024-10-11 10:55:06 gpg-agent[8602] ssh handler 0x74787b400640 for fd 107 terminated
2024-10-11 10:55:06 gpg-agent[8602] failed to build S-Exp (off=990): Cannot allocate memory
2024-10-11 10:55:06 gpg-agent[8602] failed to read the secret key

I have pinentry-program /usr/bin/pinentry-x11 so that it prompts me graphically. I always select "Save to password manager", but this doesn't seem to do anything - it requests the passphrases repeatidly.

Of course I also set a long expiry time:

default-cache-ttl 60480000
default-cache-ttl-ssh 60480000
max-cache-ttl 60480000
max-cache-ttl-ssh 60480000

I have had some MTU issues (same as git ls-remote hangs when using GIT_SSH) I've tried to address, but I don't think those are relevant here. Those were simply freezing the connection to github it self.