1

This is driving me crazy as I cannot load certain HTTPS web sites only from KVM virtual machines and only over IPv6. IPv4 works fine. IPv6 connectivity works for the same websites from the hypervisor.

My setup

  • The KVM hypervisor is running on Ubuntu 14.04.5 LTS.
  • eth0 is added to the br0 bridge interface and I use this bridge to connect the VMs to the outside world.
  • Two VMs are running on the hypervisor. The first is running on Ubuntu 12.04 (I know it has reached EOL, but that's not of concern), and the second is an Ubuntu 16.04. Both VMs experience the problem.
  • The VMs are using a Virtio interface to connect to the network.
  • IPv6 addresses are obtained by both the hypervisor and the VMs.
  • My DNS server is returning IPv6 addresses if supported by a domain, otherwise it works with IPv4.
  • I have no firewall (ip6tables) for IPv6 neither to the hypervisor nor the VMs.

    # ip6tables -v -L -n 
    Chain INPUT (policy ACCEPT 196K packets, 32M bytes)
    pkts bytes target     prot opt in     out     source               destination         
    
    Chain FORWARD (policy ACCEPT 5007K packets, 3858M bytes)
    pkts bytes target     prot opt in     out     source               destination         
    
    Chain OUTPUT (policy ACCEPT 185K packets, 30M bytes)
    pkts bytes target     prot opt in     out     source               destination         
    
    
    # ip6tables -v -L -n -t nat
    Chain PREROUTING (policy ACCEPT 1749 packets, 181K bytes)
    pkts bytes target     prot opt in     out     source               destination         
    
    Chain INPUT (policy ACCEPT 135 packets, 24165 bytes)
    pkts bytes target     prot opt in     out     source               destination         
    
    Chain OUTPUT (policy ACCEPT 187 packets, 27578 bytes)
    pkts bytes target     prot opt in     out     source               destination         
    
    Chain POSTROUTING (policy ACCEPT 1801 packets, 185K bytes)
    pkts bytes target     prot opt in     out     source               destination
    

The problem

  • IPv6 (and IPv4) connectivity works for all the web sites from the hypervisor (that's fine and as expected).

    # wget https://lwn.net -O - > /dev/null; echo Exit code: $?
    --2017-08-02 18:55:47--  https://lwn.net/
    Resolving lwn.net (lwn.net)... 2600:3c03::f03c:91ff:fe61:5c5b, 45.33.94.129
    Connecting to lwn.net (lwn.net)|2600:3c03::f03c:91ff:fe61:5c5b|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 25202 (25K) [text/html]
    Saving to: ‘STDOUT’
    
    100%[=====================================>] 25,202       149KB/s   in 0.2s   
    
    2017-08-02 18:55:48 (149 KB/s) - written to stdout [25202/25202]
    
    Exit code: 0
    
  • IPv6 connectivity works for most web sites I have tried from inside the VMs, but not all. For instance, https://lwn.net and https://hioa.no are two https web sites that I experience problems with. As you can see from the wget command below, the connection reaches a connected state but it gets stuck there:

    # wget https://lwn.net -O - > /dev/null; echo Exit code: $?
    --2017-08-02 18:53:40--  https://lwn.net/
    Resolving lwn.net (lwn.net)... 2600:3c03::f03c:91ff:fe61:5c5b, 45.33.94.129
    Connecting to lwn.net (lwn.net)|2600:3c03::f03c:91ff:fe61:5c5b|:443... connected.
    

What I have tried to troubleshoot the problem so far

  1. Started with ping6. Interestingly, pings from the VMs are working for all the domains when using IPv6! Including the ones that https is not working.

    # ping6 -c 1 -n hioa.no 
    PING hioa.no(2001:700:700:2::65) 56 data bytes
    64 bytes from 2001:700:700:2::65: icmp_seq=1 ttl=53 time=88.7 ms
    
    # ping6 -c 1 -n lwn.net
    PING lwn.net(2600:3c03::f03c:91ff:fe61:5c5b) 56 data bytes
    64 bytes from 2600:3c03::f03c:91ff:fe61:5c5b: icmp_seq=1 ttl=54 time=145 ms
    
  2. I tried to change the virtual network devices from virtio to e1000. Problem still exists.

  3. Tried to connect with IPv4 to the websites that I encounter the problem with.

    # dig A lwn.net
    
    ; <<>> DiG 9.10.3-P4-Ubuntu <<>> A lwn.net
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41423
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 4096
    ;; QUESTION SECTION:
    ;lwn.net.                       IN      A
    
    ;; ANSWER SECTION:
    lwn.net.                2633    IN      A       45.33.94.129
    

    IPv4 connectivity works fine!

    # wget --no-check-certificate https://45.33.94.129 -O - > /dev/null; echo Exit code: $?
    --2017-08-02 18:41:32--  https://45.33.94.129/
    Connecting to 45.33.94.129:443... connected.
        WARNING: certificate common name `*.lwn.net' doesn't match requested host name `45.33.94.129'.
    HTTP request sent, awaiting response... 200 OK
    Length: 25226 (25K) [text/html]
    Saving to: `STDOUT'
    
    100%[==================================>] 25,226       137K/s   in 0.2s    
    
    2017-08-02 18:41:33 (137 KB/s) - written to stdout [25226/25226]
    
    Exit code: 0
    
  4. Tried to use "openssl s_client" to connect and see if there are any error messages, but "openssl s_client" doesn't support IPv6 yet (at least not in the openssl version that is included in Ubuntu 16.04).

  5. Checked dmesg and /var/log/syslog but there is nothing related there.

Anyone has an idea of why do I get this strange behavior with some websites? Any directions on what I should try to investigate next?

  • 2
    What is the mtu for all relevant interfaces on both the VMs and the host? – user4556274 Aug 2 '17 at 16:43
  • The default. 1500 bytes. – Vangelis Tasoulas Aug 2 '17 at 16:51
2

Problem solved by reducing the MTU to 1492 in the VMs. The hypervisor is responsible to establish a PPPoE connection to the internet, and the ppp0 interface has an MTU of 1492 bytes.

Still, why would MTU be a problem since both IPv4 and IPv6 implement path MTU discovery? So why path MTU discovery is not working in this case (only for some IPv6 destinations)?

It seems like I encounter a black hole situation here.

I captured some traffic with tcpdump and loaded the file in Wireshark. I observed that the connection goes through the TCP three-way handshake as you can see in the attached picture (packet 1-3). That's also obvious from the wget output in my question where as you can see wget gets stuck after it has printed a connected message. After the successful three-way handshake the client (my VM) sends an SSL "Client Hello" message but never receives a "Server Hello" back. What the client receives is a packet which is obviously out of order based on the TCP sequence number (wireshark also reports [TCP Previous segment not captured], Continuation Data). The client then responds with an ACK (packet 6) for the last in-order packet that has been received (a duplicate ACK) and the connection stops since the server tries to resend the lost packet which is bigger than the supported MTU and never arrives. So the connection gets stuck there until I press Ctrl+C where the connection termination is initiated (packets 8-10).

Wireshark capture

Then why the Path MTU discovery is not only working for some IPv6 destinations (not all) but there is no issue with IPv4 at all? For that question, and since my installation has no IPv6 firewall in place, I assume that there is some firewall on the way towards certain web sites that blocks the ICMPv6 Packet Too Big Messages that are needed for the path MTU discovery to work. The interesting thing though is that simple ICMPv6 ping packets go through and I even receive replies.

  • 1
    In normal usage, ICMPv6 packets would never exceed 1492 octets, so wouldn't be affected by the restriction of the PPPoE link. I expect you would see ICMPv6 echo failure if you played with the packet size: ping -6 -s 1493. I have experienced similar problems with both PPPoE links and IPv6-over-IPv4 tunnels, which is what I was trying to get at with my earlier comment. Unfortunately, not directed enough, and I was too busy to come back and flesh out my question at the time. – user4556274 Aug 6 '17 at 23:07
  • That makes sense. But if we assume there is no firewall on the way that drops the ICMPv6 Packet Too Big Messages, then why the path MTU discovery mechanism is not working? – Vangelis Tasoulas Aug 7 '17 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.