0

I've got two Docker containers running (Ubuntu 22.04). They're on the same network so they can talk to each other (I've verified this with nc). In the first container, I have a program running that has created a tun device, given it an IPv4 address, and brought it up. ip addr shows

4: tun0: <POINTOPOINT,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none 
    inet 5.6.7.8/32 scope global tun0
       valid_lft forever preferred_lft forever
89: eth0@if90: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:30:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.48.2/20 brd 192.168.63.255 scope global eth0
       valid_lft forever preferred_lft forever

My program then writes a TCP SYN packet to the tun device with a source IP of 5.6.7.8 and a destination IP of 192.168.48.3 (the other container). However, the packet never reaches the other container. I've run tcpdump and the packet appears on the tun0 interface but not on eth0. tcpdump on the other container shows no packets arriving.

Why wouldn't the packet get routed through eth0?

When I started my containers via docker-compose, I set

sysctls:
  - net.ipv4.ip_forward=1

for the first container.

I've also set up NAT:

iptables -I FORWARD -i tun0 -o eth0 -j ACCEPT
iptables -I FORWARD -i eth0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE

Here's how to recreate what I'm doing:

Dockerfile:

FROM ubuntu:22.04

RUN apt -y update && \
    apt -y install build-essential iptables tcpdump

WORKDIR /app 

emit.c:

#include <arpa/inet.h>
#include <fcntl.h>
#include <linux/if_tun.h>
#include <net/if.h>
#include <netinet/in.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>

#define TUN_DEVICE "tun0"

#define SOURCE_IP "5.6.7.8"
#define DESTINATION_IP "192.168.48.3" // change to the actual address

unsigned char packet[] = {
    0x45, 0x00, 0x00, 0x54, 0x3a, 0x58, 0x40, 0x00, 0x40, 0x01, 0x03, 0x9d,
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x48, 0xc1,
    0x00, 0x03, 0x00, 0x01, 0xa1, 0x81, 0x3a, 0x65, 0x00, 0x00, 0x00, 0x00,
    0x0d, 0x81, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x11, 0x12, 0x13,
    0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
    0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b,
    0x2c, 0x2d, 0x2e, 0x2f, 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37
};

/*
Note: The IPv4 checksum will be wrong.  Run this program once and cause it to emit
the packet by sending the process SIGUSR1.  Capture the packet with tcpdump which
will tell you that the checksum is wrong and what it should be.  The result goes
at offset 10 in the packet.
*/

int tun_fd = -1;

void signal_handler(int signum)
{
    (void)signum;

    dprintf(STDOUT_FILENO, "Emitting packet\n");

    if ( write(tun_fd, packet, sizeof(packet)) < 0 ) {
        _exit(1);
    }
}


int set_address_and_bring_up(void)
{
    int ret = -1, sock;
    struct ifreq ifr = {.ifr_name = TUN_DEVICE};
    struct sockaddr_in addr = {.sin_family = AF_INET};

    inet_pton(AF_INET, SOURCE_IP, &addr.sin_addr);
    memcpy(&ifr.ifr_addr, &addr, sizeof(addr));

    sock = socket(AF_INET, SOCK_DGRAM, 0);
    if ( sock < 0 ) {
        perror("socket");
        return -1;
    }

    if ( ioctl(sock, SIOCSIFADDR, &ifr) == -1 ) {
        perror("ioctl (SIOCSIFADDR)");
        goto done;
    }

    ifr.ifr_flags = IFF_UP;
    if ( ioctl(sock, SIOCSIFFLAGS, &ifr) == -1 ) {
        perror("ioctl (SIOCSIFFLAGS)");
        goto done;
    }

    ret = 0;

done:
    close(sock);
    return ret;
}

int tun_device_create(void) {
    int fd;
    struct ifreq ifr = {.ifr_name = TUN_DEVICE, .ifr_flags = IFF_TUN | IFF_NO_PI};

    fd = open("/dev/net/tun", O_RDWR);
    if ( fd < 0 ) {
        perror("open");
        return -1;
    }

    if ( ioctl(fd, TUNSETIFF, &ifr) == -1 ) {
        perror("ioctl (TUNSETIFF)");
        goto error;
    }

    if ( set_address_and_bring_up() != 0 ) {
        goto error;
    }

    return fd;

error:
    close(fd);
    return -1;
}

int main() {
    struct sigaction action = {.sa_handler = signal_handler};

    inet_pton(AF_INET, SOURCE_IP, packet + 12);
    inet_pton(AF_INET, DESTINATION_IP, packet + 16);

    tun_fd = tun_device_create();
    if ( tun_fd < 0 ) {
        return 1;
    }

    sigaction(SIGUSR1, &action, NULL);

    while (1) {
        pause();
    }

    return 0;
}

iptables_script.sh:

#!/bin/sh -ex

iptables -I FORWARD -i tun0 -o eth0 -j ACCEPT
iptables -I FORWARD -i eth0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE

Starting containers:

# First container
docker run --rm -it --cap-add NET_ADMIN -v $PWD:/app --device=/dev/net/tun my_image bash

# Second container
docker run --rm -it ubuntu:22.04 bash

Run in first container:

test "`cat /proc/sys/net/ipv4/ip_forward`" -eq 1
gcc emit.c
./a.out &
./iptables_script.sh
kill -s USR1 `pgrep a.out`
4
  • Is it possible for you to try this without docker? Do you have two different computers (or VMs) that you can use and take Docker out of the equation for a minute?
    – ckhan
    Oct 26, 2023 at 7:24
  • @ckhan, done. I created two Ubuntu VM's on my Macbook. Same behavior. Oct 26, 2023 at 15:48
  • The VM's are run through VirtualBox and are on the same internal network. Oct 26, 2023 at 15:59
  • I've created a full MRE at this repo. Oct 26, 2023 at 18:32

1 Answer 1

0

Via pwru, I was able to discern that the packet was being rejected by fib_validate_source in the kernel as a Martian packet.

What I gathered happened was that, since the source IP of the packet was the same as that of the tun device, the packet looked suspect. Imagine if you had a physical interface with an address of 192.168.0.1 and you received a packet on that interface with a source IP of 192.168.0.1. You'd be wondering why you were sending messages to yourself.

So, I adjusted the packet's source IP to 5.6.7.9 (keeping the address of the tun device) and presto! ICMP reply received!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .