CODEWITHSUNDEEP
linuxdebianlxc
NewUser
NewUser

Reputation: 89

LXD/LXC container has no external network access (debian 12)

I am trying to setup an LXC container with communication to external network (ex: debian update servers), but like many many posts online it is not working for me and have tried all of the troubleshooting steps I could find. I am using a bare-metal Debian machine (not Proxmox or anything like that). I am using the debian/bookworm/cloud image for the container. I am open to using other debian ones but that seemed like a fine one.

My container can ping the host, but can ping no other machines on the local network and no external servers by IP address (ex: 8.8.8.8 or 1.1.1.1) which rules out DNS issues I guess.

Some example related posts researched so far (I also looked at a bunch of Proxmox ones but they may be less relevant because I am not using Proxmox).

Note, I DO have both LXC and Docker installed on the same host (just my laptop for now as I am experimenting). There are work arounds for this in the LXD documentation (other than uninstalling docker).

I enabled IPv4 forwarding as per instructions.

echo "net.ipv4.conf.all.forwarding=1" > /etc/sysctl.d/99-forwarding.conf

And I tried enabling the firewall rules with

iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT
iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

I also tried both a network bridge and a routed network port on the container hoping that one of the two would work, but both have same behavior (I set the above firewall rules for the routed network port as well: veth57853336)

LXC list

+--------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
|     NAME     |  STATE  |         IPV4          |                     IPV6                      |   TYPE    | SNAPSHOTS |
+--------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| lxc-aptcache | RUNNING | 192.168.50.150 (eth1) | fd42:1f6c:bcfe:6579:216:3eff:fe0e:8665 (eth0) | CONTAINER | 0         |
|              |         | 10.6.154.187 (eth0)   |                                               |           |           |
+--------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+

Ping from container

Can reach the host (192.168.50.66) but not anything else (ex: 192.168.50.1)

root@lxc-aptcache:~# ping 1.1.1.1
ping: connect: Network is unreachable
root@lxc-aptcache:~# ping 192.168.50.66
ping: connect: Network is unreachable
root@lxc-aptcache:~# ping -I eth1 192.168.50.66
PING 192.168.50.66 (192.168.50.66) from 192.168.50.150 eth1: 56(84) bytes of data.
64 bytes from 192.168.50.66: icmp_seq=1 ttl=64 time=0.057 ms
64 bytes from 192.168.50.66: icmp_seq=2 ttl=64 time=0.065 ms
^C
--- 192.168.50.66 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.057/0.061/0.065/0.004 ms
root@lxc-aptcache:~# 
root@lxc-aptcache:~# ping -I eth0 192.168.50.66
PING 192.168.50.66 (192.168.50.66) from 10.6.154.187 eth0: 56(84) bytes of data.
64 bytes from 192.168.50.66: icmp_seq=1 ttl=64 time=0.077 ms
64 bytes from 192.168.50.66: icmp_seq=2 ttl=64 time=0.061 ms
^C
--- 192.168.50.66 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1032ms
rtt min/avg/max/mdev = 0.061/0.069/0.077/0.008 ms
root@lxc-aptcache:~# ping -I eth0 192.168.50.1
PING 192.168.50.1 (192.168.50.1) from 10.6.154.187 eth0: 56(84) bytes of data.
^C
--- 192.168.50.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4102ms
pipe 4
root@lxc-aptcache:~# ping -I eth1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.50.150 eth1: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4095ms
pipe 4
root@lxc-aptcache:~# ping -I eth0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.6.154.187 eth0: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8200ms
pipe 4
root@lxc-aptcache:~#

Host Pings

Host can reach all of the container's failed locations but can't ping into the container.

hostUser@t440s:~$ ping 192.168.50.150
PING 192.168.50.150 (192.168.50.150) 56(84) bytes of data.
^C
--- 192.168.50.150 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4085ms

hostUser@t440s:~$ ping 10.6.154.187
PING 10.6.154.187 (10.6.154.187) 56(84) bytes of data.
^C
--- 10.6.154.187 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7172ms

hostUser@t440s:~$ ping 192.168.50.1
PING 192.168.50.1 (192.168.50.1) 56(84) bytes of data.
64 bytes from 192.168.50.1: icmp_seq=1 ttl=64 time=4.18 ms
64 bytes from 192.168.50.1: icmp_seq=2 ttl=64 time=4.64 ms
64 bytes from 192.168.50.1: icmp_seq=3 ttl=64 time=5.20 ms
^C
--- 192.168.50.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.182/4.671/5.196/0.414 ms

Investigating network links (host)

The network links appear fine on the host. But honestly, I don't understand the output of ip r very well.

hostUser@t440s:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 28:d2:44:42:4b:2d brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e0:d4:e8:53:ba:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.66/24 brd 192.168.50.255 scope global dynamic noprefixroute wlp3s0
       valid_lft 84816sec preferred_lft 84816sec
    inet6 fe80::e2d4:e8ff:fe53:ba47/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:79:d3:15:b2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:d1:b4:ef brd ff:ff:ff:ff:ff:ff
    inet 10.6.154.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:1f6c:bcfe:6579::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fed1:b4ef/64 scope link 
       valid_lft forever preferred_lft forever
18: veth39933fbb@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 3a:24:94:fb:08:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0
20: veth57853336@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a6:31:fb:a8:d4:88 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.0.1/32 scope global veth57853336
       valid_lft forever preferred_lft forever
    inet6 fe80::a431:fbff:fea8:d488/64 scope link 
       valid_lft forever preferred_lft forever

hostUser@t440s:~$ ip r
default via 192.168.50.1 dev wlp3s0 proto dhcp src 192.168.50.66 metric 600 
10.6.154.0/24 dev lxdbr0 proto kernel scope link src 10.6.154.1 
169.254.0.0/16 dev wlp3s0 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.50.0/24 dev wlp3s0 proto kernel scope link src 192.168.50.66 metric 600 
192.168.50.150 dev veth57853336 scope link

Network addresses and links on container

The container seems more suspicious since ip r is empty, but I don't know why.

root@lxc-aptcache:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:0e:86:65 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.6.154.187/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd42:1f6c:bcfe:6579:216:3eff:fe0e:8665/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe0e:8665/64 scope link 
       valid_lft forever preferred_lft forever
19: eth1@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:76:75:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.50.150/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe76:7587/64 scope link 
       valid_lft forever preferred_lft forever
root@lxc-aptcache:~# ip r
root@lxc-aptcache:~# ip r
root@lxc-aptcache:~#

More troubleshooting

These pings cross over interfaces seem to work completely fine from the host using both the bridged ETH0 interface and the routed ETH1 interface from the container. So I don't think it can be a "true" networking issue per se. I have a hypothesis it is the firewall blocking.

hostUser@t440s:~$ ip -4 route get 8.8.8.8 from 192.168.50.150 iif veth57853336
8.8.8.8 from 192.168.50.150 via 192.168.50.1 dev wlp3s0 
    cache iif veth57853336 
hostUser@t440s:~$ ip -4 route get 8.8.8.8 from 192.168.50.150 iif lxdbr0
8.8.8.8 from 192.168.50.150 via 192.168.50.1 dev wlp3s0 
    cache iif lxdbr0 
hostUser@t440s:~$ ip -4 route get 8.8.8.8 from 192.168.50.1 iif lxdbr0
8.8.8.8 from 192.168.50.1 via 192.168.50.1 dev wlp3s0 
    cache iif lxdbr0 
hostUser@t440s:~$ ip -4 route get 8.8.8.8 from 10.5.154.187 iif lxdbr0
8.8.8.8 from 10.5.154.187 via 192.168.50.1 dev wlp3s0 
    cache iif lxdbr0 
hostUser@t440s:~$

Firewall maybe?

I admit, I know very little about how iptables works. I tried dropping all rules and rerunning the above firewall allow lists but no help even after rebooting. I guess my next research area if SO community doesn't have an answer is to learn what these firewall settings mean at all.

hostUser@t440s:~$ sudo iptables -L
[sudo] password for hostUser: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (0 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (0 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-2 (0 references)
target     prot opt source               destination         

Chain DOCKER-USER (0 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain ts-forward (0 references)
target     prot opt source               destination         

Chain ts-input (0 references)
target     prot opt source               destination

LXC Config

Keeping the best for last, here is the LXC networking config used for the container.

        version: 2
        ethernets:
            eth0:
                addresses:
                  - "10.6.154.187/32"
                nameservers:
                    addresses:
                      - 1.1.1.1
                    search: []
                routes:
                  - to: 0.0.0.0/0
                    via: 169.254.0.1
                    on-link: true
            eth1:
                addresses:
                  - "192.168.50.150/32"
                nameservers:
                    addresses:
                      - 1.1.1.1
                    search: []
                routes:
                  - to: 0.0.0.0/0
                    via: 169.254.0.1
                    on-link: true
    description: Default LXD X11 profile
    devices:
      X0:
        bind: container
        connect: unix:@/tmp/.X11-unix/X0
        listen: unix:@/tmp/.X11-unix/X0
        type: proxy
      eth0:
        name: eth0
        network: lxdbr0
        type: nic
      eth1:
        ipv4.address: "192.168.50.150"
        name: eth1
        nictype: routed
        parent: wlp3s0
        type: nic
      mygpu:
        type: gpu
      root:
        path: /
        pool: default
        type: disk

And showing the bridge interface

hostUser@t440s:~$ lxc network show lxdbr0
config:
  ipv4.address: 10.6.154.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:1f6c:bcfe:6579::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/lxc-aptcache
- /1.0/profiles/ansible-profile-lxc-aptcache
- /1.0/profiles/default
- /1.0/profiles/x11
managed: true
status: Created
locations:
- none
hostUser@t440s:~$

Upvotes: 0

Views: 1277

Answers (1)

Monaco
Monaco

Reputation: 51

I have also encountered this problem, which is due to some reasons causing issues with the forwarding of iptable requests on the host. I try to add the forwarding outbound rule of lxdbr0 like that:

iptables -t nat -A POSTROUTING -i lxdbr0 -j MASQUERADE

However, it failed to work. After my attempts, the following solution ultimately helped me solve the problem:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE 
iptables -A FORWARD -i lxdbr0 -o eth0 -j ACCEPT 
iptables -A FORWARD -i eth0 -o lxdbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT

where, "lxdbr0" is the virtual interface connected to the LXC container, "eth0" is the network port on the host. These three commands allow and manage the traffic and reverse traffic from the LXC container to the external network.

For those host machine is managed using nftables, the rules for adding are as follows:

nft add rule ip nat POSTROUTING oifname "eth0" masquerade
nft add rule ip filter FORWARD iifname "lxdbr0" oifname "eth0" accept
nft add rule ip filter FORWARD iifname "eth0" oifname "lxdbr0" ct state related,established accept

The final nft list ruleset of my host like that:

for table ip nat:enter image description here

for table ip filter:enter image description here

p.s. the network port on my host is named "eno1".

Upvotes: 3

Related Questions