Connection reset by peer errror when delaying network response packets

Question

Client's side code written in redis-plus-plus

int main(){
    auto redis = Redis("tcp://192.168.122.50:6379);
    sleep(5);
    int ep = 100;

    while(true){
    cout<<"===================="<<ep<<"======================\n";

    auto pipe = redis.pipeline(false);
    for(int i=1; i<=500; i++){
        string s = to_string(i);

        if(i%2 == 1){
           pipe.set(s, s);
        }
        else {
           string st = to_string(i-1);
           pipe.get(st);
        }
    }

    auto pipe_replies = pipe.exec();
    pipe.discard();


   }

Setup:

created a network namespace.
created a veth pair vetha and vethb with vethb end in the namespace and other in the host.
added an ip 192.168.122.50 to the vethb
Started a redis server in a container in that network namespace

I'm delaying/buffering the response packets generated by the server for some dynamic time for every request at the network layer using kernel utility called plugging and unplugging qdiscs, as explained in this, this, this and this StackOverflow answer explanation (implementation is mostly similar to this one),

Tcpdump tcp stream (large file so showed only last part of it)

    secondaryvm.47504 > 192.168.122.50.6379: Flags [.], cksum 0x75ce (incorrect -> 0x9bef), seq 13086, ack 3446, win 501, options [nop,nop,TS val 3412015645 ecr 1985908099], length 0
20:18:59.436951 IP (tos 0x0, ttl 64, id 39970, offset 0, flags [DF], proto TCP (6), length 60)
    secondaryvm.47514 > 192.168.122.50.6379: Flags [S], cksum 0x75d6 (incorrect -> 0x4109), seq 995787521, win 64240, options [mss 1460,sackOK,TS val 3412015645 ecr 0,nop,wscale 7], length 0
20:18:59.436979 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.122.50.6379 > secondaryvm.47514: Flags [S.], cksum 0x75d6 (incorrect -> 0x5c7b), seq 1690924621, ack 995787522, win 65160, options [mss 1460,sackOK,TS val 1985908591 ecr 3412015645,nop,wscale 7], length 0
20:19:00.837172 IP (tos 0x0, ttl 64, id 39971, offset 0, flags [DF], proto TCP (6), length 52)
    secondaryvm.47514 > 192.168.122.50.6379: Flags [.], cksum 0x75ce (incorrect -> 0x85ee), seq 1, ack 1, win 502, options [nop,nop,TS val 3412016137 ecr 1985908591], length 0
20:19:00.837175 IP (tos 0x0, ttl 64, id 346, offset 0, flags [DF], proto TCP (6), length 52)
    secondaryvm.47504 > 192.168.122.50.6379: Flags [F.], cksum 0x75ce (incorrect -> 0x9a02), seq 13086, ack 3446, win 501, options [nop,nop,TS val 3412016137 ecr 1985908099], length 0
20:19:00.837176 IP (tos 0x0, ttl 64, id 39972, offset 0, flags [DF], proto TCP (6), length 2948)
    secondaryvm.47514 > 192.168.122.50.6379: Flags [P.], cksum 0x811e (incorrect -> 0x7451), seq 1:2897, ack 1, win 502, options [nop,nop,TS val 3412016138 ecr 1985908591], length 2896: RESP "SET" "1" "1" "GET" "1" "SET" "3" "3" "GET" "3" "SET" "5" "5" "GET" "5" "SET" "7" "7" "GET" "7" "SET" "9" "9" "GET" "9" "SET" "11" "11" "GET" "11" "SET" "13" "13" "GET" "13" "SET" "15" "15" "GET" "15" "SET" "17" "17" "GET" "17" "SET" "19" "19" "GET" "19" "SET" "21" "21" "GET" "21" "SET" "23" "23" "GET" "23" "SET" "25" "25" "GET" "25" "SET" "27" "27" "GET" "27" "SET" "29" "29" "GET" "29" "SET" "31" "31" "GET" "31" "SET" "33" "33" "GET" "33" "SET" "35" "35" "GET" "35" "SET" "37" "37" "GET" "37" "SET" "39" "39" "GET" "39" "SET" "41" "41" "GET" "41" "SET" "43" "43" "GET" "43" "SET" "45" "45" "GET" "45" "SET" "47" "47" "GET" "47" "SET" "49" "49" "GET" "49" "SET" "51" "51" "GET" "51" "SET" "53" "53" "GET" "53" "SET" "55" "55" "GET" "55" "SET" "57" "57" "GET" "57" "SET" "59" "59" "GET" "59" "SET" "61" "61" "GET" "61" "SET" "63" "63" "GET" "63" "SET" "65" "65" "GET" "65" "SET" "67" "67" "GET" "67" "SET" "69" "69" "GET" "69" "SET" "71" "71" "GET" "71" "SET" "73" "73" "GET" "73" "SET" "75" "75" "GET" "75" "SET" "77" "77" "GET" "77" "SET" "79" "79" "GET" "79" "SET" "81" "81" "GET" "81" "SET" "83" "83" "GET" "83" "SET" "85" "85" "GET" "85" "SET" "87" "87" "GET" "87" "SET" "89" "89" "GET" "89" "SET" "91" "91" "GET" "91" "SET" "93" "93" "GET" "93" "SET" "95" "95" "GET" "95" "SET" "97" "97" "GET" "97" "SET" "99" "99" "GET" "99" "SET" "101" "101" "GET" "101" "SET" "103" "103" "GET" "103" "SET" "105" "105" "GET" "105" "SET" "107" "107" "GET" "107" "SET" "109" "109" "GET" "109" "SET" "111" "111" "GET" "111" "SET" "113" "113" "GET" "113" "SET" "115" "115" [|RESP]
20:19:00.837177 IP (tos 0x0, ttl 64, id 39974, offset 0, flags [DF], proto TCP (6), length 2948)
    secondaryvm.47514 > 192.168.122.50.6379: Flags [P.], cksum 0x811e (incorrect -> 0x6f2d), seq 2897:5793, ack 1, win 502, options [nop,nop,TS val 3412016138 ecr 1985908591], length 2896: RESP "ET" "115" "SET" "117" "117" "GET" "117" "SET" "119" "119" "GET" "119" "SET" "121" "121" "GET" "121" "SET" "123" "123" "GET" "123" "SET" "125" "125" "GET" "125" "SET" "127" "127" "GET" "127" "SET" "129" "129" "GET" "129" "SET" "131" "131" "GET" "131" "SET" "133" "133" "GET" "133" "SET" "135" "135" "GET" "135" "SET" "137" "137" "GET" "137" "SET" "139" "139" "GET" "139" "SET" "141" "141" "GET" "141" "SET" "143" "143" "GET" "143" "SET" "145" "145" "GET" "145" "SET" "147" "147" "GET" "147" "SET" "149" "149" "GET" "149" "SET" "151" "151" "GET" "151" "SET" "153" "153" "GET" "153" "SET" "155" "155" "GET" "155" "SET" "157" "157" "GET" "157" "SET" "159" "159" "GET" "159" "SET" "161" "161" "GET" "161" "SET" "163" "163" "GET" "163" "SET" "165" "165" "GET" "165" "SET" "167" "167" "GET" "167" "SET" "169" "169" "GET" "169" "SET" "171" "171" "GET" "171" "SET" "173" "173" "GET" "173" "SET" "175" "175" "GET" "175" "SET" "177" "177" "GET" "177" "SET" "179" "179" "GET" "179" "SET" "181" "181" "GET" "181" "SET" "183" "183" "GET" "183" "SET" "185" "185" "GET" "185" "SET" "187" "187" "GET" "187" "SET" "189" "189" "GET" "189" "SET" "191" "191" "GET" "191" "SET" "193" "193" "GET" "193" "SET" "195" "195" "GET" "195" "SET" "197" "197" "GET" "197" "SET" "199" "199" "GET" "199" "SET" "201" "201" "GET" "201" "SET" "203" "203" "GET" "203" "SET" "205" "205" "GET" "205" "SET" "207" "207" "GET" "207" "SET" "209" "209" "GET" "209" "SET" "211" "211" "GET" "211" "SET" "213" "213" "GET" "213" "SET" "215" "215" "GET" "215" "SET" "217" "217" "GET" "217" "SET" "219" "219" "GET" "219" "SET" "221" "221" "GET" "221" "SET" "223" "223" "GET" "223" "SET" "225" [|RESP]
20:19:00.837213 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.122.50.6379 > secondaryvm.47514: Flags [R], cksum 0x8c9b (correct), seq 1690924622, win 0, length 0
20:19:00.837220 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.122.50.6379 > secondaryvm.47514: Flags [R], cksum 0x8c9b (correct), seq 1690924622, win 0, length 0
20:19:00.837223 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.122.50.6379 > secondaryvm.47514: Flags [R], cksum 0x8c9b (correct), seq 1690924622, win 0, length 0
20:19:00.837527 IP (tos 0x0, ttl 64, id 48265, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.50.6379 > secondaryvm.47504: Flags [F.], cksum 0x75ce (incorrect -> 0x929c), seq 3446, ack 13087, win 501, options [nop,nop,TS val 1985909992 ecr 3412016137], length 0
20:19:01.395138 IP (tos 0x0, ttl 64, id 347, offset 0, flags [DF], proto TCP (6), length 52)
    secondaryvm.47504 > 192.168.122.50.6379: Flags [.], cksum 0x75ce (incorrect -> 0x8d23), seq 13087, ack 3447, win 501, options [nop,nop,TS val 3412017538 ecr 1985909992], length 0
20:19:01.395141 IP (tos 0x0, ttl 64, id 15348, offset 0, flags [DF], proto TCP (6), length 60)
    secondaryvm.47520 > 192.168.122.50.6379: Flags [S], cksum 0x75d6 (incorrect -> 0xd113), seq 3516159171, win 64240, options [mss 1460,sackOK,TS val 3412017680 ecr 0,nop,wscale 7], length 0
20:19:01.395187 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.122.50.6379 > secondaryvm.47520: Flags [S.], cksum 0x75d6 (incorrect -> 0xc191), seq 3522767979, ack 3516159172, win 65160, options [mss 1460,sackOK,TS val 1985910549 ecr 3412017680,nop,wscale 7], length 0
20:19:02.077257 IP (tos 0x0, ttl 64, id 15349, offset 0, flags [DF], proto TCP (6), length 52)
    secondaryvm.47520 > 192.168.122.50.6379: Flags [.], cksum 0x75ce (incorrect -> 0xeb51), seq 1, ack 1, win 502, options [nop,nop,TS val 3412018095 ecr 1985910549], length 0
20:19:11.230107 IP (tos 0x0, ttl 64, id 15350, offset 0, flags [DF], proto TCP (6), length 52)
    secondaryvm.47520 > 192.168.122.50.6379: Flags [F.], cksum 0x75ce (incorrect -> 0xc96a), seq 1, ack 1, win 502, options [nop,nop,TS val 3412026773 ecr 1985910549], length 0
20:19:11.230196 IP (tos 0x0, ttl 64, id 60988, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.122.50.6379 > secondaryvm.47520: Flags [F.], cksum 0x75ce (incorrect -> 0xa2f6), seq 1, ack 2, win 510, options [nop,nop,TS val 1985920384 ecr 3412026773], length 0
20:19:11.922439 IP (tos 0x0, ttl 64, id 15351, offset 0, flags [DF], proto TCP (6), length 52)
    secondaryvm.47520 > 192.168.122.50.6379: Flags [.], cksum 0x75ce (incorrect -> 0x9e79), seq 2, ack 2, win 502, options [nop,nop,TS val 3412027930 ecr 1985920384], length 0
20:28:47.589154 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::188c:7fff:fe8b:4de0 > ip6-allrouters: [icmp6 sum ok] ICMP6, router solicitation, length 16
      source link-address option (1), length 8 (1): 1a:8c:7f:8b:4d:e0
        0x0000:  1a8c 7f8b 4de0
20:30:58.660993 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) secondaryvm > ip6-allrouters: [icmp6 sum ok] ICMP6, router solicitation, length 16
      source link-address option (1), length 8 (1): 82:8b:f2:84:7f:bc
        0x0000:  828b f284 7fbc
20:35:16.865946 IP6 (flowlabel 0xd8c02, hlim 255, next-header UDP (17) payload length: 149) secondaryvm.mdns > ff02::fb.mdns: [bad udp cksum 0xf0f1 -> 0x2942!] 0 [9q] PTR (QM)? _nfs._tcp.local. PTR (QM)? _ipp._tcp.local. PTR (QM)? _ipps._tcp.local. PTR (QM)? _ftp._tcp.local. PTR (QM)? _webdav._tcp.local. PTR (QM)? _webdavs._tcp.local. PTR (QM)? _sftp-ssh._tcp.local. PTR (QM)? _smb._tcp.local. PTR (QM)? _afpovertcp._tcp.local. (141)

The problem is Client get this error after some time of execution:

Failed to get reply: Connection reset by peer

How to prevent this error?

Redis version 6.0.9 and client is running directly on the host.

Guess: Your "dynamic delay time" messed up packet order so much that some part in the network stack decided "this is completely broken" and reset the connection. To verify, you'd have to be able to reproduce the delay; and you could also try to identify the ordering that caused the problem (use Wireshark for this, not tcpdump). You can also experiment with less delay and delay ranges causing less or no out-of-order packets. — dirkt, Dec 29 '20 at 8:03

Stack Exchange Network

current community

your communities

more stack exchange communities

Connection reset by peer errror when delaying network response packets

0

Your Answer

Browse other questions tagged kernel-modules tcpdump tc network-namespaces redis or ask your own question.

Linked

Hot Network Questions

Connection reset by peer errror when delaying network response packets

0

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.

Your Answer

Sign up or log in

Post as a guest

Browse other questions tagged kernel-modules tcpdump tc network-namespaces redis or ask your own question.

Linked

Related

Hot Network Questions