3

I have two namespaces srv1 and srv2, interconnected via a softswitch (p4 bmv2) with veth pairs. The softswitch does just simple forwarding. The veth interfaces inside the namespaces have IP addresses assigned to them (respectively 192.168.1.1 and 192.168.1.2). I could ping between the two namespaces using those IP addresses:

sudo ip netns exec srv1 ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=1.04 ms

but when I try netcat I don't receive messages on the server side:

client:

sudo ip netns exec srv1 netcat 192.168.1.2 80 -u
 hello!

server:

sudo ip netns exec srv2 netcat -l 80 -u

The interface receives the packets with proper format. I verified with tcpdump on both namespaces and I saw the packets being sent and received properly:

client:

sudo ip netns exec srv1  tcpdump -XXvv -i srv1p 
tcpdump: listening on srv1p, link-type EN10MB (Ethernet), capture size 262144 bytes
^C06:09:41.088601 IP (tos 0x0, ttl 64, id 14169, offset 0, flags [DF], proto UDP (17), length 35)
    192.168.1.1.55080 > 192.168.1.2.http: [bad udp cksum 0x8374 -> 0x5710!] UDP, length 7
    0x0000:  00aa bbcc dd02 00aa bbcc dd01 0800 4500  ..............E.
    0x0010:  0023 3759 4000 4011 801d c0a8 0101 c0a8  .#7Y@.@.........
    0x0020:  0102 d728 0050 000f 8374 6865 6c6c 6f21  ...(.P...thello!
    0x0030:  0a                                       .

1 packet captured
1 packet received by filter
0 packets dropped by kernel

server:

sudo ip netns exec srv2  tcpdump -XXvv -i srv2p
tcpdump: listening on srv2p, link-type EN10MB (Ethernet), capture size 262144 bytes
^C06:09:41.089232 IP (tos 0x0, ttl 64, id 14169, offset 0, flags [DF], proto UDP (17), length 35)
    192.168.1.1.55080 > 192.168.1.2.http: [bad udp cksum 0x8374 -> 0x5710!] UDP, length 7
    0x0000:  00aa bbcc dd02 00aa bbcc dd01 0800 4500  ..............E.
    0x0010:  0023 3759 4000 4011 801d c0a8 0101 c0a8  .#7Y@.@.........
    0x0020:  0102 d728 0050 000f 8374 6865 6c6c 6f21  ...(.P...thello!
    0x0030:  0a                                       .

1 packet captured
1 packet received by filter
0 packets dropped by kernel

I added on srv2 iptable rules to ACCEPT udp packets on port 80 and to LOG:

sudo ip netns exec srv2 iptables -t filter -A INPUT -p udp --dport 80 -j ACCEPT
sudo ip netns exec srv2 iptables -I INPUT -p udp --dport 80 -j LOG --log-prefix " IPTABLES " --log-level=debug

I could see the stats increasing on the entry and the packets being logged on /var/log/kern.log, but the message never reaches the netcat's listening socket.

sudo ip netns exec srv2 iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    33 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:80 LOG flags 0 level 7 prefix " IPTABLES "
    4   133 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:80

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 

kernel logs:

kernel: [581970.306032]  IPTABLES IN=srv2p OUT= MAC=00:aa:bb:cc:dd:02:00:aa:bb:cc:dd:01:08:00 SRC=192.168.1.1 DST=192.168.1.2 LEN=33 TOS=0x00 PREC=0x00 TTL=64 ID=51034 DF PROTO=UDP SPT=48784 DPT=80 LEN=13 

When I replace the softswitch with a bridge, the netcat works. I thought maybe the softswitch corrupts the packets but the tcpdump shows the right format. The UDP checksum is not correct but it is generated like that from the source server, and it is the same thing when using the linux bridge anyways but it works in that case. Is there a way to know the reason those packets do not reach the netcat server?

EDIT: Here are more informations requested in the comments:

$ sudo ip netns exec srv1 ip -br link
lo               DOWN           00:00:00:00:00:00 <LOOPBACK> 
srv1p@if2948     UP             00:aa:bb:cc:dd:01 <BROADCAST,MULTICAST,UP,LOWER_UP> 

$ sudo ip netns exec srv2 ip -br link
lo               DOWN           00:00:00:00:00:00 <LOOPBACK> 
srv2p@if2944     UP             00:aa:bb:cc:dd:02 <BROADCAST,MULTICAST,UP,LOWER_UP>



$ sudo ip netns exec srv1 ip route
192.168.1.0/24 dev srv1p  proto kernel  scope link  src 192.168.1.1 

$ sudo ip netns exec srv2 ip route
192.168.1.0/24 dev srv2p  proto kernel  scope link  src 192.168.1.2 

$ sudo ip netns exec srv2 ip neighbour
192.168.1.7 dev srv2p lladdr 00:aa:bb:cc:dd:0d STALE
192.168.1.4 dev srv2p lladdr 00:aa:bb:cc:dd:06 STALE
192.168.1.8 dev srv2p lladdr 00:aa:bb:cc:dd:0e STALE
192.168.1.6 dev srv2p lladdr 00:aa:bb:cc:dd:0a STALE
192.168.1.1 dev srv2p lladdr 00:aa:bb:cc:dd:01 STALE
192.168.1.3 dev srv2p lladdr 00:aa:bb:cc:dd:05 STALE
192.168.1.5 dev srv2p lladdr 00:aa:bb:cc:dd:09 STALE

$ sudo ip netns exec srv1 ip neighbour
192.168.1.8 dev srv1p lladdr 00:aa:bb:cc:dd:0e STALE
192.168.1.4 dev srv1p lladdr 00:aa:bb:cc:dd:06 STALE
192.168.1.2 dev srv1p lladdr 00:aa:bb:cc:dd:02 STALE
192.168.1.7 dev srv1p lladdr 00:aa:bb:cc:dd:0d STALE
192.168.1.5 dev srv1p lladdr 00:aa:bb:cc:dd:09 STALE
192.168.1.3 dev srv1p lladdr 00:aa:bb:cc:dd:05 STALE
192.168.1.6 dev srv1p lladdr 00:aa:bb:cc:dd:0a STALE

$ /sbin/sysctl -ar '\.rp_filter'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.docker0.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.veth0b5517e.rp_filter = 2
net.ipv4.conf.veth5c69974.rp_filter = 2
net.ipv4.conf.veth6c63fa9.rp_filter = 2
net.ipv4.conf.veth87b4442.rp_filter = 2
net.ipv4.conf.vethaecf041.rp_filter = 2
net.ipv4.conf.vethc9014e5.rp_filter = 2
net.ipv4.conf.vethf280de4.rp_filter = 2
net.ipv4.conf.vethf442944.rp_filter = 2
2
  • Just in case: You need to start the netcat server before the netcat client (but you probably did, judging from the dump). I usually start an xterm in each namespace, and su it to my normal user; that makes debugging so much easier. Also try a non-system port instead of 80, e.g. 2345, just in case something on port 80 gets in the way. – dirkt Jun 28 '20 at 16:05
  • I tried with ports other that 80 and it is always the same results. I edited the question to add the results of the commands you requested – user3228522 Jun 30 '20 at 19:45
0

The issue was related to udp checksum. The packets are generated by netcat with bad UDP checksum and a flag is set in the kernel to ignore it. It works with linux bridge because the packets stay in the kernel space, but when they pass through the softswitch the packets enter the user space and are reinsterted in the kernel without the inital flags that ignore bad checksum, so they get dropped.

That was fixed by checksum offload on the interfaces:

/sbin/ethtool --offload $intf tx off
/sbin/ethtool --offload $intf rx off

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.