1

I have docker running on a synology nas. The nas has a private network (172.17.0.0/16) where docker places the containers. It also has a public facing interface on my lan, call it 10.11.12.10/24.

                   internet
                      ^
                      |
                      |
                 +----+------+
                 |  pfSense  |
                 +-----+-----+
                       |
                 +-----+-----+
                 |   switch  |
                 +-+-------+-+
                   |       |
                   |       |
               +---+--+  +-+---+
      +------->|  AP  |  | NAS |
      |        +------+  +--+--+
      |                     |
+-----+--+               +--+--------+
| laptop |               | container |
+--------+               +-----------+

It is totally possible to set up port forwarding from the lan interface to a docker container and access ssh on a container that way, but I was feeling lazy so I thought I could just add a route in my pfsense firewall that would redirect all traffic to 172.17.0.0/16 from my lan directly to the nas.

The route seem to work, it is possible to ssh to docker containers directly from my lan. But no such ssh session can live longer than 45 seconds.

➜  ~ date; ssh 172.17.0.2 -l root
Wed Oct 10 21:19:32 CEST 2018
Last login: Wed Oct 10 19:16:07 2018 from 10.11.12.182
root@ubuntu1:~# while true; do date; sleep 5; done
Wed Oct 10 19:19:42 UTC 2018
Wed Oct 10 19:19:47 UTC 2018
Wed Oct 10 19:19:52 UTC 2018
Wed Oct 10 19:19:57 UTC 2018
Wed Oct 10 19:20:02 UTC 2018
Wed Oct 10 19:20:07 UTC 2018
packet_write_wait: Connection to 172.17.0.2 port 22: Broken pipe
➜  ~
➜  ~

The last packets before it stops seen with tcpdump on the laptop looks like this:

16:38:59.072210 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633305, win 5348, options [nop,nop,TS val 561669488 ecr 134368964], length 0                               
16:38:59.103814 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [.], seq 9633305:9634753, ack 5966, win 199, options [nop,nop,TS val 134369004 ecr 561669467], length 1448           
16:38:59.225048 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669637 ecr 134369004], length 0                               
16:38:59.324726 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369225 ecr 561669467], length 1448          
16:38:59.324789 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669736 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.035790 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369666 ecr 561669467], length 1448          
16:39:00.035854 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561670445 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.751550 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134370548 ecr 561669467], length 1448          
16:39:00.751642 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561671159 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:02.493382 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134372312 ecr 561669467], length 1448          
16:39:02.493451 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561672900 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:06.179874 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134375840 ecr 561669467], length 1448          
16:39:06.179964 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561676574 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:13.249689 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134382896 ecr 561669467], length 1448          
16:39:13.249741 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561683642 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:27.376570 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134396992 ecr 561669467], length 1448          
16:39:27.376645 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561697743 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0

Corresponding tcpdump on pfsense only see traffic from the laptop as replies does not need to be routed

16:38:59.083374 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9632853, win 5358, options [nop,nop,TS val 561669485 ecr 134368962], length 0                                 
16:38:59.083422 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9632905, win 5361, options [nop,nop,TS val 561669488 ecr 134368962], length 0                                 
16:38:59.083471 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9632957, win 5359, options [nop,nop,TS val 561669488 ecr 134368962], length 0                                 
16:38:59.083510 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633001, win 5358, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083542 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633053, win 5356, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083581 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633105, win 5355, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083680 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633157, win 5353, options [nop,nop,TS val 561669488 ecr 134368963], length 0                                 
16:38:59.083731 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633209, win 5351, options [nop,nop,TS val 561669488 ecr 134368964], length 0                                 
16:38:59.083778 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633261, win 5350, options [nop,nop,TS val 561669488 ecr 134368964], length 0                                 
16:38:59.083824 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9633305, win 5348, options [nop,nop,TS val 561669488 ecr 134368964], length 0                                 
16:38:59.228487 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669637 ecr 134369004], length 0                                 
16:38:59.328679 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561669736 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.039683 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561670445 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:00.755280 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561671159 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:02.497019 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561672900 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:06.183600 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561676574 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:13.255414 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561683642 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0
16:39:27.380228 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9634753, win 5363, options [nop,nop,TS val 561697743 ecr 134369004,nop,nop,sack 1 {9628757:9630205}], length 0

The nas sees the following

16:38:59.061977 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632365:9632409, ack 5966, win 199, options [nop,nop,TS val 134368960 ecr 561669467], length 44            
16:38:59.062027 IP 10.11.12.182.64941 > 172.17.0.2.22: Flags [.], ack 9628757, win 5360, options [nop,nop,TS val 561669467 ecr 134368941], length 0                               
16:38:59.062216 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632409:9632453, ack 5966, win 199, options [nop,nop,TS val 134368960 ecr 561669467], length 44              
16:38:59.062437 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632453:9632505, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 52              
16:38:59.062620 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632505:9632549, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 44              
16:38:59.062837 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632549:9632601, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 52              
16:38:59.062984 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632601:9632653, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 52              
16:38:59.063200 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632653:9632697, ack 5966, win 199, options [nop,nop,TS val 134368961 ecr 561669467], length 44              
16:38:59.063401 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632697:9632749, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.063605 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632749:9632801, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.063800 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632801:9632853, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.064021 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632853:9632905, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.064220 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632905:9632957, ack 5966, win 199, options [nop,nop,TS val 134368962 ecr 561669467], length 52              
16:38:59.064447 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9632957:9633001, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 44              
16:38:59.064698 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633001:9633053, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 52              
16:38:59.064938 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633053:9633105, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 52              
16:38:59.065172 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633105:9633157, ack 5966, win 199, options [nop,nop,TS val 134368963 ecr 561669467], length 52              
16:38:59.065422 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633157:9633209, ack 5966, win 199, options [nop,nop,TS val 134368964 ecr 561669467], length 52              
16:38:59.065663 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633209:9633261, ack 5966, win 199, options [nop,nop,TS val 134368964 ecr 561669467], length 52              
16:38:59.065880 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9633261:9633305, ack 5966, win 199, options [nop,nop,TS val 134368964 ecr 561669467], length 44              
16:38:59.105416 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [.], seq 9633305:9634753, ack 5966, win 199, options [nop,nop,TS val 134369004 ecr 561669467], length 1448             
16:38:59.326452 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369225 ecr 561669467], length 1448            
16:38:59.767432 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134369666 ecr 561669467], length 1448            
16:39:00.649466 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134370548 ecr 561669467], length 1448            
16:39:02.413454 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134372312 ecr 561669467], length 1448            
16:39:05.941490 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134375840 ecr 561669467], length 1448            
16:39:12.997468 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134382896 ecr 561669467], length 1448            
16:39:27.093471 IP 172.17.0.2.22 > 10.11.12.182.64941: Flags [P.], seq 9628757:9630205, ack 5966, win 199, options [nop,nop,TS val 134396992 ecr 561669467], length 1448          

I'm not sure I read it right, but the last few packages have the same ack/seq values. Leading me to believe that both nodes see each others packets but are unable to realise that the packets are part of the same session.

Any idea?

4
  • 1
    When you run wireshark or tcpdump on the various involved devices, where does the traffic stop getting sent? Commented Oct 10, 2018 at 21:50
  • I honestly can't tell. I've started tcpdump on laptop, pfsense and nas, and they all seem to stop in unison.
    – azzid
    Commented Oct 11, 2018 at 17:43
  • What happens if you ping your laptop from the container? Is there a chance they you're missing a route somewhere?
    – V13
    Commented Oct 11, 2018 at 18:18
  • ping works, but is stateless
    – azzid
    Commented Oct 11, 2018 at 18:35

1 Answer 1

1

Adding the route to the pfsense makes the traffic flow weird.

Traffic from the laptop bounces over the pfsense, back down to the nas and then to the container. Replies from the container however will go via the nas directly to the laptop on layer 2. No routing needed.

A consequence of that is that the firewall on the nas that only allow related packages get confused and invalidates the relation between the ssh packages and the first ssh connect packet.

My guess is that iptables conntrack on the nas is killing the session.

# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DEFAULT_FORWARD
-N DOCKER
-N DOCKER-ISOLATION
-A FORWARD -j DEFAULT_FORWARD
-A DEFAULT_FORWARD -j DOCKER-ISOLATION
-A DEFAULT_FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DEFAULT_FORWARD -o docker0 -j DOCKER
-A DEFAULT_FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A DEFAULT_FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN

Removing the route from the pfsense and configuring it directly on the laptop instead seem to work a lot better.

How to add route on macOS:

sudo route add 172.17.0.0/16 10.11.12.10
2
  • Since the Synology is doing NAT, there's probably an async traffic pattern somewhere, because the synology would see the source IP of the laptop as being on the same local LAN. laptop -> pfsense forward rules -> synology NAT -> container -> Synology (NAT table lookup) -> local LAN -> laptop. I can't figure out why it's working for the first 45 seconds, though. In that situation, it makes perfect sense that a local route for the container subnet pointing to the synology as the gateway for that network would work. Commented Oct 11, 2018 at 18:38
  • NAT would mask the 172 adress consistently. I do see that adress outsider docker. I think iptables is the culprit.
    – azzid
    Commented Oct 11, 2018 at 18:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .