2

When you want to mark a packet in iptables, you would generally add the following line to your firewall script:

iptables -t mangle -A POSTROUTING -p tcp -m multiport --dports 80,443 -j MARK --set-mark 2

I know this subject is a little bit complicated, but let's focus just on the single rule. When you add something like the one above, in the /proc/net/nf_conntrack file some entries will have the mark set:

ipv4     2 tcp      6 3706 ESTABLISHED ... mark=2

The problem with the rule is that when you add another rule that matches, for instance source or destination address and set another mark, the previous mark will be rewritten to something you set. But there are some ways to "add" the marks. So if one rule set mark=2 and another rule set mark=5, then the resulting mark will be mark=7, or something like that.

I have one working example based on mwan3 , but I don't really get it. I know what the mangle table looks like after starting the tool, and what rules were added:

enter image description here

So to understand the mechanism, I have to know what actually happens to the arriving packet. But in this example, the marking rules are different.

We have two different WAN interfaces. Based on the marks, the packets will go to different routing tables. So what actually happens to a packet that is destined to port 443 and, for example, to port 1000? Could anyone help me analyze the rules?

5
+50

The target "-j MARK --set-mark 2" will set the mark 2 on the packet, whatever the previous value was. If you want to avoid your mark to be erased, you can simply end the packet path in the chain with -j ACCEPT. For example :

iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j MARK --set-mark 10
iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j ACCEPT
iptables -t mangle -A POSTROUTING -d 8.8.8.8 -j MARK --set-mark 20
iptables -t mangle -A POSTROUTING -d 8.8.8.8 --dport 80 -j ACCEPT
# If you open a connection to 8.8.8.8:80, the mark will be 10

Although, you have to take care if you are in the main chain (eg. POSTROUTING) or in a custom chain: ACCEPT will end the main chain, while RETURN will end the current chain. It depends on your needs.

About the mwan3 example, it is quite hard to be sure without the filter, nat, and raw table, and without tc configuration.

However, it looks like this:

  • The "mwan3_rules" chain sends packets from a new connection with dport 80 or 443 to the "mwan3_policy_balanced" chain
  • The "mwan3_policy_balanced" chain set the mark 200 on 40% of the packets, and the mark 100 on the other ones
  • The "mwan3_rules" chain sends packets from a new connection not with dport 80 or 443 to the "mwan3_policy_wan_only" chain
  • The "mwan3_policy_wan_only" chain set the mark 100
  • I guess packet with mark 100 will go through cable, while packets with mark 200 will go through lte

So 40% of http(s) traffic will go through lte, 60% through cable, and everything else through cable.

If your goal is to loadbalance connections on your 2 ISP, you should probably write your own iptables rules from scratch, since mwan3 ones are hard to read.

You could start with this question.

Good luck !

Edition:

The documentation states:

--set-xmark value[/mask]
    Zero out the bits given by mask and XOR value into the ctmark.

If you have a 0x100 mark, and try to set xmark 0x200/0xff00:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 1111 0000 0000 --> 0000 0000 0000 0000
  • XOR value into the ctmark: 0000 0000 0000 0000 XOR 0000 0010 0000 0000 --> 0000 0010 0000 0000 --> 0x200 --> 512

If you have a 0x100 mark, and try to set xmark 0x200/0xf000:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 0000 0000 0000 --> 0000 0001 0000 0000
  • XOR value into the ctmark: 0000 0001 0000 0000 XOR 0000 0010 0000 0000 --> 0000 0011 0000 0000 --> 0x300 --> 768

If you have a 0x100 mark, and try to set xmark 0x100/0xf000:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 0000 0000 0000 --> 0000 0001 0000 0000
  • XOR value into the ctmark: 0000 0001 0000 0000 XOR 0000 0001 0000 0000 --> 0000 0000 0000 0000 --> 0x000 --> 0

If you have a 0x100 mark, and try to set xmark 0x100/0xff00:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 1111 0000 0000 --> 0000 0000 0000 0000
  • XOR value into the ctmark: 0000 0000 0000 0000 XOR 0000 0001 0000 0000 --> 0000 0001 0000 0000 --> 0x100 --> 256

In the mwan3 file the case is always this one:

  • You have a 0x0 mark, and try to set xmark 0x??00/0xff00
  • In the ctmark, clear the bits given by the mask: 0000 0000 0000 0000 AND NOT 1111 1111 0000 0000 --> 0000 0000 0000 0000
  • XOR value into the ctmark: 0000 0000 0000 0000 XOR ???? ???? 0000 0000 --> ???? ???? 0000 0000 --> 0x??00 --> ?
  • With this mask and these values, set-xmark just replace the previous value

Now, let's go through the chains:

  • Prerouting jumps to mwan3_hook.
  • mwan3_hook restore connmarks into marks
  • mwan3_hook sends the new connections (mark = 0x0) to mwan3_ifaces
  • mwan3_ifaces sends the new connections to mwan3_iface_wan
  • mwan3_iface_wan set the mark 0xff00 to the new connections from interface eth0, when the source address is in the ipset list mwan3_connected
  • mwan3_iface_wan set the mark 0x100 to the other new connections on interface eth0
  • mwan3_ifaces sends the new connections to mwan3_iface_lte
  • mwan3_iface_lte set the mark 0xff00 to the new connections from interface wwan, when the source address is in the ipset list mwan3_connected
  • mwan3_iface_lte set the mark 0x200 to the other new connections on interface wwan
  • Note : at this point, all incoming connections are marked
  • mwan3_hook sends the connections to mwan3_connected
  • mwan3_connected set the mark 0xff00 when the destination address is in the ipset list mwan3_connected
  • mwan3_hook sends the connections to mwan3_track
  • mwan3_track set the mark 0xff00 to the connection if the destination ip is in the ipset list mwan3_track_wan, the packet is an icmp echo of 32 bytes length
  • mwan3_track set the mark 0xff00 to the connection if the destination ip is in the ipset list mwan3_track_lte, the packet is an icmp echo of 32 bytes length
  • mwan3_hook sends the connections to mwan3_rules
  • mwan3_rules sends the new tcp/80 or tcp/443 connections from the inside to mwan3_policy_balanced
  • mwan3_policy_balanced set the mark 0x200 to 40% of the new connection
  • mwan3_policy_balanced set the mark 0x100 to the other new connection
  • mwan3_hook restore marks into connmarks
  • mwan3_hook set the mark 0xff00 to the connections still not marked (from the inside, not icmp type 8, tcp/80, or tcp/443)
  • Prerouting jumps to fwmark, but the chain is not on the screen
  • Forward fix the mss
  • Output jumps to mwan3_hook, and every steps are done again.

In the end, we have 3 states:

  • 0x100 mark (256) : connections from wan, and 60% of http(s) connections to the internet
  • 0x200 mark (512) : connections from lte, and 40% of http(s) connections to the internet
  • 0xff00 mark (65280) : other traffic

Since we haven't the ip rule, we can only guess:

  • 0x100 mark will go through wan routing table
  • 0x200 mark will go through lte routing table
  • 0xff00 will go through another routing table
  • The mark in the packets has to be rewritten. mwan3 uses only the rules you see on the photo. It's not that hard, I just don't understand how the marking mechanism with masks works. So that's why I need a step by step explanation. What actually happens to a mark of a packet when the packet passes the rules. – Mikhail Morfikov May 16 '16 at 21:52
  • I updated the answer to explain xmarks with masks, and walkthrough the chains. – Sanael May 17 '16 at 15:29
  • Could you tell me what AND NOT means? – Mikhail Morfikov May 17 '16 at 16:38
  • 1
    Sure. It's actually A and (not B). Not means 1 becomes 0 and 0 become 1. So if B is 0101, (not B) is 1010. The and part now: you compare bit per bit, and you need both sides to be 1. So if you have 0011 and 1010, result is 0010. – Sanael May 17 '16 at 16:46
  • There are some bugs in the answer. For instance, the mark 0x100/0xff00 doesn't actually replace the previous value. It replaces only the affected bits by the mask. So if you had a packet that is marked already as 0x4, this mark "survives" the mwan3markings. I mean, the mark will be added, and in result you get 0x104, which is 260 in decimal instead of 256. The same with chains -- not mark = 0x0, just mark=0x0000/0xff00, you can still have a mark 0x4, and it will match the rule. Also avoid using add, I think it should be set instead. – Mikhail Morfikov May 17 '16 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.