10

I have a test suite which initializes a server on the localhost and then runs thousands of stress tests which include starting and stopping clients connecting to this local server on same machine. I am trying to find the reason for a RST sent by client to the server.

This is the sequence of events:

  1. client initializes SYN
  2. Server sends back SYN,ACK
  3. client sends RST (instead of ACK and establishing connection)

One observation is on many occasions servers runs around 99 percent CPU although it returns back to normal in few seconds/minutes.

Initially I thought this could be because client is using a port from previously closed connection which is in TIME_WAIT state and so when server sends a SYN,ACK on that connection client issues RST. Although I was skeptical as to why client should use a port in TIME_WAIT state but then I am using old fedora core 4 and thought may be it doesn't have proper implementation of TCP/IP :-).

Now I don't believe that is the reason because all the tuning done around fixing above issue did not help alleviate the problem. FYI following are the changes I tried in /etc/sysctl.conf:

net.ipv4.ip_local_port_range = 1024     65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_tw_reuse = 1

One more point, this issue isn't consistent. In fact I can reproduce this problem only after I go thru 15 to 35 cycles of Re-starting the server and running all client tests.

I ran out of all the reasons I can think of why this could be happening so let me know if anyone has any ideas on what might be happening.

Packet details

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378209 1032719 1858.494749 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=0 WS=128

Frame 1032719: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378209000 EDT
    Epoch Time: 1403722185.378209000 seconds
    [Time delta from previous captured frame: 0.005893000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 1858.494749000 seconds]
    Frame Number: 1032719
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf489 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378222 1032720 1858.494762 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 navisphere > netview-aix-5 [SYN, ACK] Seq=0 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=1587252 WS=128

Frame 1032720: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378222000 EDT
    Epoch Time: 1403722185.378222000 seconds
    [Time delta from previous captured frame: 0.000013000 seconds]
    [Time delta from previous displayed frame: 0.000013000 seconds]
    [Time since reference or first frame: 1858.494762000 seconds]
    Frame Number: 1032720
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 0, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0xf7fa [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 1587252
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032719]
        [The RTT to ACK the segment was: 0.000013000 seconds]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378228 1032721 1858.494768 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032721: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378228000 EDT
    Epoch Time: 1403722185.378228000 seconds
    [Time delta from previous captured frame: 0.000006000 seconds]
    [Time delta from previous displayed frame: 0.000006000 seconds]
    [Time since reference or first frame: 1858.494768000 seconds]
    Frame Number: 1032721
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375927 1032722 1861.492467 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=0 WS=128

Frame 1032722: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375927000 EDT
    Epoch Time: 1403722188.375927000 seconds
    [Time delta from previous captured frame: 2.997699000 seconds]
    [Time delta from previous displayed frame: 2.997699000 seconds]
    [Time since reference or first frame: 1861.492467000 seconds]
    Frame Number: 1032722
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf19b [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375950 1032723 1861.492490 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 [TCP Previous segment lost] navisphere > netview-aix-5 [SYN, ACK] Seq=2997725 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=1588002 WS=128

Frame 1032723: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375950000 EDT
    Epoch Time: 1403722188.375950000 seconds
    [Time delta from previous captured frame: 0.000023000 seconds]
    [Time delta from previous displayed frame: 0.000023000 seconds]
    [Time since reference or first frame: 1861.492490000 seconds]
    Frame Number: 1032723
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: Bad TCP]
    [Coloring Rule String: tcp.analysis.flags]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 2997725, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 2997725    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0x3414 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 1588002
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032722]
        [The RTT to ACK the segment was: 0.000023000 seconds]
        [TCP Analysis Flags]
            [A segment before this frame was lost]
                [Expert Info (Warn/Sequence): Previous segment lost (common at capture start)]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375958 1032724 1861.492498 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032724: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375958000 EDT
    Epoch Time: 1403722188.375958000 seconds
    [Time delta from previous captured frame: 0.000008000 seconds]
    [Time delta from previous displayed frame: 0.000008000 seconds]
    [Time since reference or first frame: 1861.492498000 seconds]
    Frame Number: 1032724
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:54.376072 1032725 1867.492612 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1589502 TSecr=0 WS=128

Frame 1032725: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:54.376072000 EDT
    Epoch Time: 1403722194.376072000 seconds
    [Time delta from previous captured frame: 6.000114000 seconds]
    [Time delta from previous displayed frame: 6.000114000 seconds]
    [Time since reference or first frame: 1867.492612000 seconds]
    Frame Number: 1032725
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xebbf [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1589502, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

iptables-save output:

# Generated by iptables-save v1.3.0 on Thu Jun 26 10:15:33 2014
*filter
:FORWARD ACCEPT [0:0]
:INPUT ACCEPT [675:236200]
:OUTPUT ACCEPT [241:21540]
COMMIT
# Completed on Thu Jun 26 10:15:33 2014
Improve this question
9
  • Is the socket from the client in non-blocking mode? Is it possible that client socket be closed by the time the server accepts? Do you have timings for those packets? A pcap maybe?
    – Stéphane Chazelas
    Jun 25, 2014 at 21:25
  • 1
    Do you check the return value of the connect()s? Is it possible that you do two non-blocking connect()s on the same socket? Is it possible that some clients get killed or interrupted in some way (alarm()?) in the middle of a connect()? Can you reproduce it with the client running under strace -ff or autrace?
    – Stéphane Chazelas
    Jun 26, 2014 at 8:48
  • 1
    Anything fancy (nfqueue, connection limiting...) in the output of iptables-save? Possibility that some resource limit be reached? How many concurrent clients? Do those bind to specific source port?
    – Stéphane Chazelas
    Jun 26, 2014 at 8:55
  • 1
    Would be better added to the question. The question is more whether those limits are reached (or likely reached) in your test than what they actually are.
    – Stéphane Chazelas
    Jun 26, 2014 at 15:42
  • 1
    My guess would be that the stress put on the stack has one connection fall in timeout client side at half open state. That connection is dropped (which means that the client app should see it as a ETIMEDOUT), and when the syn/ack is finally processed by the stack, there are no more connections to relate it to, and thus it gets reset. Did you try raising the size of the syn backlog (sysctl -w net.ipv4.tcp_max_syn_backlog=???)?
    – didierc
    Jun 27, 2014 at 23:58

1 Answer 1

Reset to default
1

Just want to conclude this thread with adopted solution in absence of a fix to the root cause which I am guessing is in the Fedora 4 TCP stack implementation. As a solution I simply introduced an immediate re-connect attempt in case the failure is due to ETIMEDOUT and protocol used is IPV6. This has fixed the problem for me and my team for good with possible risk of additional 3 min delay in reporting error in any other connection issue resulting in ETIMEDOUT. This is not the real/ideal solution to the problem, but gets us around to move on... as its only impacts our automated test suite and not getting shipped to customer. I hope eventually somebody who knows fedora 4 tcp/ip stack implementation well enough will solve this mystery for good.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.