1

I would like to study the behaviour of an application streaming from a slow server. I'm trying to take advantage of tc-netem to introduce a network delay.

Before setting up a more complicated scenario, I decided to do this on a virtual machine, which is supposed to emulate the slow server. I access the VM via ssh, so I decided to create a virtual ethernet which will be delayed, while I plan to use the real ethernet device for management.

First I created the fake interface with ip link add link eth0 type macvtap and I assign it with an IP address.

Then I added a 40 milliseconds delay with tc qdisc add dev macvtap0 root netem delay 40000. This effectively allowed me to have a throughput drop (from ~250 MiB/sec to ~6 MiB/sec).

At this point I tried to play a bit with my setup, and I started to realize that the delay was not just affecting the macvtap0 device, but also the eth0 device I was connecting through (in short, my ssh session started to lag).

I think that my netem delay affected the actual NIC. Is this due to the fact I'm working within a VM? Or should I have used something different tan macvtap? Or could it be because I applied my changes to the root qdisc?

EDIT - This is the first time for me entering the apparently huge world of traffic differentiation. Perhaps there's a better approach to this? E.g. can I set up a queue to slow down a selected process? I decided to rename this question to reflect my actual purpose.

0
1

A remark first: adding latency isn't the same as limiting bandwidth. Consider geosync satellite internet link: there's a mandatory latency (due to speed of light) of ~ 1/4s for one-way stream, so possibly ~1/2s round-trip if the whole path is via satellite, and it's said to be able to reach 506 Mbps with modern technology: a lot of data can be on the way in this 1/2s window, it doesn't have to wait for each packet's ack before sending the next. Of course latency will not help speed and can hinder it because of waiting acknowledgements, needing retransmission etc.

As the issue for which the question was asked and the goal of the question are really different, I'll only answer about the issue below beside these informations.

Some qdisc have built-in features for rate limiting: htb, tbf, ... Here are a two pointers on limiting bandwidth (rather than adding latency):

  • Network Bandwidth Limiting on Linux with TC

    # tc qdisc add dev eth0 root tbf rate 1024kbit latency 50ms burst 1540
    

    Note that as usual, the limitation is on egress. To limit on ingress, an IFB device must be inserted, as the example in my answer to Simulation of packet loss on bridged interface using netem .

  • Good (not so) old wondershaper script (v1.4. It's outdated in some distributions like Debian), which can do wonders: while designed to prioritise flows and lower their latency, it's good as well to lower rate, and makes already use of IFB to limit in a simple way both egress and ingress.

    # wondershaper -a eth0 -d 1000 -u 100 would limit eth0's bandwidth to 1000 kbps download and 100 kbps upload.

Now about the issue...


The issue is not related to macvtap (where the tap part isn't even used in OP's setting), nor macvlan nor any virtualization nor tc-netem. It's related to the way routing works when multiple interfaces are put in the same ethernet LAN using the same IP LAN.

Since there is no way to know for sure what happened without OP's specific configuration in the VM, I'll reproduce an artificial example in a few network namespaces: "system" peer pinging "system" twocards, both linked via "switch" bridge.

Initial configuration before adding the 2nd card:

ip netns del bridge || :
ip netns del twocards || :
ip netns del peer || :
ip netns add bridge
ip netns add twocards
ip netns add peer
ip -n bridge link add bridge0 type bridge
ip -n twocards link add eth0 type veth peer netns bridge name port1
ip -n peer link add eth0 type veth peer netns bridge name port2
ip -n bridge link set port1 master bridge0
ip -n bridge link set port2 master bridge0
ip -n bridge link set port1 up
ip -n bridge link set port2 up
ip -n bridge link set bridge0 up
ip -n twocards link set eth0 up
ip -n peer link set eth0 up
ip -n twocards address add dev eth0 192.0.2.1/24
ip -n peer address add dev eth0 192.0.2.2/24

Now:

# ip -n twocards route
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.1 

On a typical installation using the ubiquitous NetworkManager, there would be a metric 100 set for an ethernet device, let's do the same:

ip -n twocards route add 192.0.2.0/24 dev eth0 src 192.0.2.1 metric 100
ip -n twocards route del 192.0.2.0/24 dev eth0 src 192.0.2.1

Let's measure:

# ip netns exec peer ping -c2 192.0.2.1 
PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data.
64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=0.116 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=64 time=0.060 ms

--- 192.0.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.060/0.088/0.116/0.028 ms

Normal result. Add the 2nd card, put an IP on it and add the netem queue:

ip -n twocards link add eth1 type veth peer netns bridge name port3
ip -n bridge link set port3 master bridge0
ip -n bridge link set port3 up
ip -n twocards link set eth1 up
ip -n twocards address add dev eth1 192.0.2.3/24
ip netns exec twocards tc qdisc add dev eth1 root netem delay 40000

New route:

# ip -n twocards route
192.0.2.0/24 dev eth1 proto kernel scope link src 192.0.2.3 
192.0.2.0/24 dev eth0 scope link src 192.0.2.1 metric 100 

New behaviour (assuming inherited rp_filter is set):

# ip netns exec peer ping -c2 192.0.2.1 
PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data.

--- 192.0.2.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1021ms

What happened? Now the routing and ARP behaviour came into play:

# ip -n twocards route get 192.0.2.2 from 192.0.2.1
192.0.2.2 from 192.0.2.1 dev eth1 uid 0 
    cache 
# ip -n twocards -br link
lo               DOWN           00:00:00:00:00:00 <LOOPBACK> 
eth0@if3         UP             96:45:4d:f0:52:35 <BROADCAST,MULTICAST,UP,LOWER_UP> 
eth1@if5         UP             c2:70:b7:40:6c:40 <BROADCAST,MULTICAST,UP,LOWER_UP> 

# ip -n peer neighbour
192.0.2.1 dev eth0 lladdr 96:45:4d:f0:52:35 REACHABLE

The routing table on twocards would now use eth1, even for the address assigned to eth0. Linux just followed its routing tables default behaviour. Because rp_filter kicked in, ping wasn't accepted when received on device eth0 (whose MAC is still in peer's cache). If rp_filter hadn't been set, this would have worked with an asymmetric route: ping request on eth0, ping reply on eth1, (delayed by netem) until peer's ARP cache eventually expired, then using eth1 only (with only egress delayed as usual).

Let's clear ARP cache on peer and try again:

# ip -n peer neighbour flush dev eth0
# ip netns exec peer ping -c2 192.0.2.1 
PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data.
64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=80.2 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=64 time=40.1 ms

--- 192.0.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 40.141/60.178/80.215/20.037 ms

# ip -n peer neighbour
192.0.2.1 dev eth0 lladdr c2:70:b7:40:6c:40 REACHABLE

The first ping (only) is delayed twice by netem: once for the underlying ARP reply to the ARP request, once for the actual ping reply.

Let's remove the extra device's route on twocards and clear again the ARP cache on peer:

ip -n twocards route del 192.0.2.0/24 dev eth1
ip -n peer neighbour flush dev eth0

Normal result again:

# ip netns exec peer ping -c2 192.0.2.1
PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data.
64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=0.108 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=64 time=0.050 ms

--- 192.0.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.050/0.079/0.108/0.029 ms

Likewise pinging the IP assigned to eth1 will now go through twocards's eth0 without delay:

# ip netns exec peer ping -c2 192.0.2.3
PING 192.0.2.3 (192.0.2.3) 56(84) bytes of data.
64 bytes from 192.0.2.3: icmp_seq=1 ttl=64 time=0.143 ms
64 bytes from 192.0.2.3: icmp_seq=2 ttl=64 time=0.042 ms

--- 192.0.2.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.042/0.092/0.143/0.051 ms

So what should be done about this? Just don't use on the same system two network devices on the same ethernet LAN, especially when using the same IP LAN, this will lead to troubles. A macvlan or macvtap device isn't usually supposed to be used directly by the host: it's supposed to be handed over to a VM or a container (an other network namespace) with its own separate routing stack, so this wouldn't be a problem in a typical usage.

If for whatever reason two cards on the same LAN must be used (outside of bonding, teaming etc.), then an additional quite complex configuration must be done to avoid this. See my answer on SF to Ghost ping on a multi-NIC Linux system for details.

0

I'm not sure about the problem you have, so I'm skipping that.

If you want to fiddle with traffic shaping per process you will need to use a classful queuing discipline. HTB or HSFC are probably your best bet. Using that you can create a tree of queueing disciplines (netem can be attached to one of the leaves) and assign traffic across them with tc filter.

Filtering is quite flexible because of the fw filter method wich can look for an iptables mark, which in turn means that you can select the traffic using iptables. You can also select the traffic directly though.

Having said that, please note that qdiscs are only effective on outgoing traffic. You can have an ingress qdisc but that's very limited and probably won't behave as you'd expect.

For testing purposes, a good bet would be to create a real VM with two interfaces and somehow force-route your traffic through that. Some tricker may be required (i.e. a couple of levels of NATing). In the VM you can then attach whatever qdisc you like on the two interfaces, controlling both directions of the traffic.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.