Container running on Podman can ping host machine's private network IP address but cannot connect to service listening on that address

Question

I have a fairly simple Dockerfile:

ARG CONTAINER_BASE_IMAGE
FROM ${CONTAINER_BASE_IMAGE}

COPY /buildenv_vars.sh /root/buildenv_vars.sh

RUN touch /root/.profile && cat /root/buildenv_vars.sh >> /root/.profile

# For debugging Ubuntu container, uncomment this line
RUN apt-get update && apt-get install -y bind9-host iputils-ping telnet && setcap cap_net_raw+p /usr/bin/ping

CMD [ \
    "/bin/bash", \
    "-l", "-c", \
    "echo TSH=$TSH; echo TBH=$TBH; echo PATH=$PATH; echo Waiting for signal...; trap \"echo Shutting down; exit 0\" SIGTERM SIGINT SIGKILL; /bin/sleep infinity & wait" \
]

In my particular case at this time, $CONTAINER_BASE_IMAGE is ubuntu:jammy. Then I have this fairly simple Compose file:

services:
  container1:
    image: "${NAT_TRAV_IMG}"
    volumes:
      - "${TSH}:${TSH}"
      - "${TBH}:${TBH}"
    ports:
      - "${NAT_TRAV_EM_PORT:-44100}:${NAT_TRAV_EM_PORT:-44100}"

  container1:
    image: "${NAT_TRAV_IMG}"
    volumes:
      - "${TSH}:${TSH}"
      - "${TBH}:${TBH}"
    ports:
      - "${NAT_TRAV_SUB_PORT:-44200}:${NAT_TRAV_SUB_PORT:-44200}"

I start the containers using podman-compose up -d, and then podman-compose ps shows this:

CONTAINER ID  IMAGE                     COMMAND               CREATED         STATUS         PORTS                     NAMES
0acc4de1b578  localhost/me/myimagename  /bin/bash -l -c e...  10 minutes ago  Up 10 minutes  0.0.0.0:44286->44286/tcp  mydirectoryname_container1_1
52d81a75c453  localhost/me/myimagename  /bin/bash -l -c e...  10 minutes ago  Up 10 minutes  0.0.0.0:44386->44386/tcp  mydirectoryname_container2_1

Furthermore, the command podman inspect -f '{{.Name}}:|:{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' shows that my container’s IP addresses are 10.89.2.2 and 10.89.2.3, respectively. So far so good. But here’s where things get a little confusing:

My host machine has the IP 172.24.52.166 on the local private network, and it has a service listening on port 46541 on that IP address (and only that IP address … not on 127.0.0.1 or anything else, and I can’t change that). My host machine’s firewall is disabled:

$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

Other machines on the 172.24.52/24 network can ping my host machine and can telnet-connect (using telnet for simplistic connectivity test) to that listening service on port 46541:

$ ping 172.24.52.166
PING 172.24.52.166 (172.24.52.166): 56 data bytes
64 bytes from 172.24.52.166: icmp_seq=0 ttl=64 time=1.073 ms
64 bytes from 172.24.52.166: icmp_seq=1 ttl=64 time=0.868 ms
64 bytes from 172.24.52.166: icmp_seq=2 ttl=64 time=0.975 ms
^C
--- 172.24.52.166 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.868/0.972/1.073/0.084 ms

$ telnet 172.24.52.166 46541
Trying 172.24.52.166...
Connected to 172.24.52.166.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

I also need my containers to be able to be able to connect to that service running on the host machine. When I “shell into” my container using (for example) the command podman exec -it mydirectoryname_container2_1 /bin/bash -l, it can ping 172.24.52.166, but it cannot telnet-connect to 172.24.52.166 on port 46541:

root@52d81a75c453:/# ping 172.24.52.166
PING 172.24.52.166 (172.24.52.166) 56(84) bytes of data.
64 bytes from 172.24.52.166: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 172.24.52.166: icmp_seq=2 ttl=64 time=0.065 ms
64 bytes from 172.24.52.166: icmp_seq=3 ttl=64 time=0.067 ms
^C
--- 172.24.52.166 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2029ms
rtt min/avg/max/mdev = 0.063/0.065/0.067/0.001 ms

root@52d81a75c453:/# telnet 172.24.52.166 46541
Trying 172.24.52.166...
telnet: Unable to connect to remote host: Connection refused

I don’t feel like I’m doing anything fancy or difficult here, so I’m not sure why I can’t connect. FWIW:

$ podman-compose --version
podman-compose version 1.2.0
podman version 5.3.1

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
...

Also, my container can access the internet just fine, and before I added the apt-get commands to the Dockerfile to streamline my debugging, I was running apt-get just fine within the running container as well. The problem appears to be limited to connecting to the host machine for something other than pinging.

I haven’t tried Docker yet. I’m probably going to try that next to see if maybe Podman is the problem, but I wanted to get this working with Podman.