bind mounts get removed with network namespaces?

Question

Strongswan daemon puts its .pid and .ctl files in /var/run to detect if it is already running.

I want to run multiple of these daemons on the same machine within different network namespaces. I am trying to achieve this by bind mounting different directories (say /etc/namespace1 to namespace1's /var/run and /etc/namespace2 to namespace2's /var/run). /var/run is a symlink to /run, so I bind mount to /run like below.

I can almost achieve this like this:

""In default namespace""

$:~ sudo echo "red" >> /etc/red/run/pidfile
$:~ sudo echo "blue" >> /etc/blue/run/pidfile
$:~ sudo ip netns exec red

""In red namespace""

$:~ mount --bind /etc/red/run/ /run/
$:~ cat /var/run/pidfile

red

""In blue namespace""

 $:~ mount --bind /etc/blue/run/ /run/
 $:~ cat /var/run/pidfile

blue

So this works fine. This way the daemon when it creates /var/run/charon.pid while inside red will not confuse with blue namespace's /var/run/charon.pid and two instances may start.

However, here's the issue: if I "exit" from red namespace and then re-enter via "ip netns exec red bash", then the mount is no longer present. That is there is no /var/run/redfile at all.

So, the question is how can I make this sticky? Do I need /etc/fstab changes? But it doesn't work. If asked, I can provide details of "it doesn't work".

I am lost. Will appreciate some help.

Thanks!

Answer 1

The simple solution would be to instruct each instance of strongswan to use a different directory to store the PID file by means of setting the correct value of the IPSEC_PIDDIR environment variable in your start and stop script.

Answer 2

ip netns exec already bind mounts /etc/netns/<netns name>/* over the corresponding file/directory in /etc. So you may compile strongSwan with e.g. --with-piddir=/etc/ipsec.d/run and then create the required directories so each instance creates its PID files in separate directories:

# mkdir -p /etc/ipsec.d/run
# mkdir -p /etc/netns/<netns name 1>/ipsec.d/run
# mkdir -p /etc/netns/<netns name 2>/ipsec.d/run

More details can be found on the strongSwan wiki.

Answer 3

Problem cause AFAIK

ip netns add ${NSNAME} creates a persistent namespace, by mounting it to /run/netns/${NSNAME}. This in itself does not involve mount namespaces, yet.

Doing ip netns exec ${NSNAME} cmd spawns a temporary mount namespace that exists until cmd exits, after that the mount namespace gets cleaned up, only network namespace remains. This is required to achieve the magic mounting of /etc/netns/${NSNAME}/*. Unfortunately any mounts done by the cmd are cleaned up too.

From man ip-netns:

ip netns exec automates handling of this configuration, file convention for network namespace unaware applications, by creating a mount name‐ space and bind mounting all of the per network namespace configure files into their traditional location in /etc.

A solution

Short version

    # Init for persistent mount namespaces
    $ mkdir /run/mntns \
        && mount --bind --make-private /run/mntns /run/mntns

    NSNAME=red

    # Create persistent network+mount namespaces
    $ ip netns add ${NSNAME} \
        && touch "/run/mntns/${NSNAME}" \
        && ip netns exec "${NSNAME}" \
            sh -x -c "nsenter -t $$ --mount mount --bind /proc/\$\$/ns/mnt '/run/mntns/${NSNAME}'"

    # Do NOT use `ip netns exec`
    # (Re)enter the namespaces with `nsenter`, mounts are persistent
    $ nsenter --net=/run/netns/${NSNAME} --mount=/run/mntns/${NSNAME} \
          program arg1 arg2

Long version

It is possible to create both network and mount namespaces persistent.

This is largely adapted from examples in the unshare man page: http://man7.org/linux/man-pages/man1/unshare.1.html

The main implication is that instead of ip netns exec the nsenter must now be used to use both the network+mount namespaces, at least where mounts matter anyway.

Step 1. Prepare. Create directory for persistent mount namespaces

This needs to be done once, e.g. after boot, NOT for every namespace. The path here is arbitrary. I'm trying to use similar naming to that used by ip netns.

    # Try to create a dir for persistent mount namespaces
    # and if it got created (e.g. first attempt since boot),
    # then bind mount it with --make-private as required by `unshare --mount=/path`
    $ mkdir /run/mntns \
        && mount --bind --make-private /run/mntns /run/mntns

Note, the mkdir ... && mount ... are not atomic and where there are multiple concurrently starting services doing this sequence then a race condition may occur.

Step 2. Create network+mount persistent namespaces

2a. method A, simple, no magic bind mounts.

This creates persistent mount+network namespaces. It is simple but does not create magic bind mounts for /etc/netns/${NSNAME}/*.

    $ NSNAME=red # give it a name

    # For network namespace use same paths as used by `ip netns` for interoperability
    # touch can fail to update timestamp if file is already mounted
    # Try to create files for persistent mounts.
    # and if touch is successful
    # then create persistent mount and network namespaces
    # Instead of `true` can run a command right away e.g. /bin/bash
    # As it is this is approximate equivalent of `ip netns add ${NANAME}`
    $ touch /run/netns/${NSNAME} /run/mntns/${NSNAME} \
        && unshare --net=/run/netns/${NSNAME} --mount=/run/mntns/${NSNAME} true

2b. method B, leverage ip netns exec to make mounts but make persistent.

To get those bind mount done by ip netns exec ... but make them persistent, can do setup as follows. Note this must be done once ONLY when creating the namespace, not every time.

    $ ip netns add ${NSNAME} \
        && touch "/run/mntns/${NSNAME}" \
        && ip netns exec "${NSNAME}" \
            sh -x -c "nsenter -t $$ --mount mount --bind /proc/\$\$/ns/mnt '/run/mntns/${NSNAME}'"

For an attempted explanation see towards the end.

Steps 3. Use it

Now the nsenter must be used to launch commands inside both the namespaces as required to achieve persistent mount results. This is regardless if setup was done with unshare or ip netns exec NAME sh ... nsneter ... mount ....

    # Get a shell inside the namespaces
    $ nsenter --net=/run/netns/${NSNAME} --mount=/run/mntns/${NSNAME} \
        /bin/bash

e.g. with OP's commands:

    # Do your additional mounts (can also be done above as command to `unshare`)
    # This is approximate equivalent of 
    #   `ip netns exec ${NANAME} mount --bind /etc/red/run/ /run/`
    $ nsenter --net=/run/netns/${NSNAME} --mount=/run/mntns/${NSNAME} \
          mount --bind /etc/${NSNAME}/run/ /run/

    # Re-enter the namespaces, check that the mounts are still there
    $ nsenter --net=/run/netns/${NSNAME} --mount=/run/mntns/${NSNAME} \
          sh -c 'ls /run/'

Important

This approach works, but instead of using ip netns exec use nsenter --net=... --mount=...! If ip netns exec ... is used it will replace (for the duration of its run) the mount namespace, which can be ok for commands that do not care about mount's persistence, e.g.

    # This does not touch any files, only operates on network device
    ip netns exec ${NSNAME} ip link set up dev lo

Step Z. - Clean up

Regardless which of the two methods above was used to create the namespaces, the clean up can be done by either of the two methods:

    # This deletes the /mnt/netns/${NSNAME} for us
    ip netns del ${NSNAME}
    # Unmount and delete mount namespace holder
    umount "/mnt/mntns/${NSNAME}" \
        && rm "/mnt/mntns/${NSNAME}"

or

    # Unmount and delete both namespaces' holders
    umount "/mnt/mntns/${NSNAME}" "/mnt/netns/${NSNAME}" \
        && rm "/mnt/mntns/${NSNAME}" "/mnt/netns/${NSNAME}"

Explanation

Apologies for long text. This is in part for my own benefit.

This command from step 2b above deserves some explanation.

The command again.

    $ ip netns add ${NSNAME} \
        && touch "/run/mntns/${NSNAME}" \
        && ip netns exec "${NSNAME}" \
            sh -x -c "nsenter -t $$ --mount mount --bind /proc/\$\$/ns/mnt '/run/mntns/${NSNAME}'"

From man unshare

The namespaces can optionally be made persistent by bind mounting /proc/pid/ns/type files to a filesystem path and entered with nsenter(1) even after the program terminates (except PID namespaces where permanently running init process is required). Once a persistent namespace is no longer needed, it can be unpersisted with umount(8).

Explanation start

Here is what it does, if broken down into simple steps. Note, two shells are required for this demo.

    # SHELL-1

    # Init
    $ mkdir /run/mntns \
        && mount --bind --make-private /run/mntns /run/mntns

    # Prep
    $ NSNAME=red

    # Create namespace
    $ ip netns add ${NSNAME}
    # Start a shell inside the namespace
    $ ip netns exec "${NSNAME}" /bin/bash

    # Now inside a shell inside a namespace.
    # Now need to make a bind mount it this shell's mount namespace
    $ echo $$ # Use this in SHELL2 AS SHELL1_PID
    # Go to SHELL-2 before exiting this one

    # SHELL-2

    # Set this to PID from SHELL-1
    $ SHELL1_PID=????

    $ NSNAME=red
    $ touch "/run/mntns/${NSNAME}"

    $ mount --bind /proc/${SHELL1_PID}/ns/mnt "/run/mntns/${NSNAME}"

Now both network+mount namespaces are persistent and can be re-entered with nsenter as shown way earlier.

The important part is mounting ip netns exec's /proc/pid/ns/net into the outer shell's mount namespace before ip netns exec.. exits.

Explanation cont.

So the answer is: from within ip netns exec ... do nsenter, but not as a direct command ip netns exec ${NSNAME} nsenter ..., because that would leave and lose the temporary namespace, but rather cause a fork() by running an intermediate shell that keeps the temporary namespace active, and the forked nsenter does mount. This following is closer to how the one long command works.

    # Init
    $ mkdir /run/mntns \
        && mount --bind --make-private /run/mntns /run/mntns

    # Prep
    $ export NSNAME=red
    $ touch "/run/mntns/${NSNAME}"

    $ export MOUNT_PARENT_PID=$$ # for simplicity

    # Create namespace
    $ ip netns add ${NSNAME}
    # Start a shell inside the namespace, the intermediate shell is important
    $ ip netns exec "${NSNAME}" /bin/bash
    # Now inside a forked namespaced shell

    # Need to mount `ip netns exec` temporary mount namespace to a file
    # within MOUNT_PARENT_PID's namespace to make it persistent
    # nsenter will reference it later

    $$ nsenter -t ${MOUNT_PARENT_PID} --mount \
           mount --bind /proc/$$/ns/mnt "/run/mntns/${NSNAME}"

    # Yes, here $MOUNT_PARENT_PID can be replaced by $PPID
    # But if any intermediate sub-shells and forks get introduced
    # then $PPID can point to a different process.

Manually mounting /etc/netns/${NSNAME}/*

The original answer had this section, so here it is, although likely not very useful.

Re-Creating /mnt/netns/${NSNAME}/* bind mounts

The downside of using unshare is you don't get the automatic bind mounting of /etc/netns/NETNS_NAME/* that is provided by ip netns exec ${NSNAME} cmd, which is not hard to add manually.

    # either during unshare or later as nsenter.
    # The important difference is here it is done once and result persists
    # unlike with `ip netns exec` which does this every time
    NSNAME="${NSNAME}" \
      nsenter --net=/run/netns/${NSNAME} --mount=/run/mntns/${NSNAME} \
      /bin/sh -e -x -c \
      'cd "/etc/netns/${NSNAME}"; for f in *; do mount --bind "/etc/netns/${NSNAME}/${f}" "/etc/${f}"; done;'