Strange network behaviour when using bridge and macvlan network together with publish port

[JSkold]

Hi,
I'm seeing some strange behaviour in containers connected to both a MACVLAN and bridge network at the same time, along with a port publish. When I run a container on both networks and publish a port via the host, any attempt to connect to that port will have responses sent with a source IP address of the internal bridge IP address on the external interface.

The above image shows a wireshark capture from my client device (192.168.1.20) connecting to the podman host (192.168.1.10) on port 5000 which is published by a container running with two podman networks, a MACVLAN and a bridge network. The IP 10.95.0.26 is the IP address the container has on the bridge network. The outgoing packets from the container are not properly masqueraded to the host IP address and my client will reject them.

All of this is with rootful containers.

My question

What is the expected behaviour?
MACVLAN and IPVLAN networks don't support port publish, but in this case I'm also using a bridge network. There is to my knowledge no way of specifying what network a port publish should affect.

What I expect to happen:
I'd expect the bridge network to work as usual regardless of if the container also is connected to a MACVLAN network. I.e. the response packets should have the IP address of the podman host.

Should this be filed as a bug, in what repository in that case?

Detailed setup and reproduction

I have a client device on a local network together with the podman host:

• Client device IP: 192.168.1.20
• Podman host IP: 192.168.1.10

On this network a DHCP server hands out IP addresses. The subnet is 192.168.1.0/24.

The MACVLAN network's parent interface is connected to the local network. Defined by the file:

[Network]
Driver=macvlan
NetworkName=macvlan
PodmanArgs=--interface-name=enp4s0
IPv6=false
IPAMDriver=dhcp

[Install]
WantedBy=network.target

The bridge network in defined by this file:

[Network]
DisableDNS=false
Driver=bridge
NetworkName=standard
IPv6=false
Subnet=10.95.0.0/16
Gateway=10.95.0.1

[Install]
WantedBy=network.target

Run a container with both networks and a port publish (I've used jonlabelle/network-tools when debugging this) and listen for TCP connections on the published port :
podman run --log-driver=k8s-file --rm -it --network standard --network macvlan -p 5000:5000 docker.io/jonlabelle/network-tools socat TCP-LISTEN:5000 STDIO

On the client device attempt to connect to the podman host on the published port:
nc 192.168.1.10 5000

Observe the behaviour I mentioned above. Incoming connections to 192.168.1.10:5000 will have responses sent with a source IP address in 10.95.0.0/16.

If I omit --network macvlan everything works as I'd expect, I can establish a connection to the container.

Podman host details

$ uname -sr
Linux 6.7.7-arch1-1

$ podman version
Client:       Podman Engine
Version:      4.9.3
API Version:  4.9.3
Go Version:   go1.22.0
Git Commit:   8d2b55ddde1bc81f43d018dfc1ac027c06b26a7f-dirty
Built:        Fri Feb 16 17:18:03 2024
OS/Arch:      linux/amd64

$ /usr/lib/podman/netavark --version
netavark 1.10.3

[JSkold]

This is not an issue with podman, it's a routing issue.

The routing table in the container with the above setup will look something like:

default via 10.95.0.1 dev eth1 proto static metric 100
default via 192.168.1.1 dev eth0 proto static metric 100
10.95.0.0/16 dev eth1 proto kernel scope link src 10.95.0.26
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.147

Since the bridge network uses DNAT the incoming packets still has source address 192.168.1.20. Response packets will then be routed via eth0 in this case, regardless of which interface they came from. I wrongly assumed that response packets would be sent back on the same interface they came from.

There are some solutions using connection marking and alternative routing tables that I'm going give a shot, described here:
https://unix.stackexchange.com/questions/693365/reply-on-same-interface-as-incoming-but-with-dynamic-ip-addresses