3

I have a linux box where we see a lot of (30%) TCP retransmission in tcpdump when receiving (downloading) files from outside. Using the dropwatch utility we see many packet drops in kernel function net_receive_skb(). That means data have been received on the NIC, but later some of them are dropped in the kernel when processing the packets. The many dropped packets can explain the necessity of retransmissions.

dropwatch output is like the following:

dropwatch -l kas 
Initalizing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
1 drops at tcp_rcv_established+906 (0xffffffff814d0a66)
6 drops at unix_dgram_connect+4ac (0xffffffff8151890c)
6 drops at unix_dgram_connect+4ac (0xffffffff8151890c)
19 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
5 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
9 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
7 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
6 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
14 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
15 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
2 drops at __netif_receive_skb+49f (0xffffffff8147b4ef)
2 drops at inet_csk_reset_xmit_timer.clone.1+265 (0xffffffff814d9cb5) ^CGot a stop message
dropwatch> exit
Shutting down ...

The system is a CentOS 6.2 with a 2.6.32 kernel (centOS package name 2.6.32-696.el6.x86_64). So I looked into the version of netif_receive_skb in kernel source code, trying to find the reason for the packet drops. And I see there is only one place calling kfree_skb (near the end of the function) that will leave a trace on dropped packets. The code is:

int netif_receive_skb(struct sk_buff *skb)
{
    struct packet_type *ptype, *pt_prev;
    struct net_device *orig_dev;
    struct net_device *null_or_orig;
    int ret = NET_RX_DROP;
    __be16 type;

    if (!skb->tstamp.tv64)
        net_timestamp(skb);

    if (skb->vlan_tci && vlan_hwaccel_do_receive(skb))
        return NET_RX_SUCCESS;

    /* if we've gotten here through NAPI, check netpoll */
    if (netpoll_receive_skb(skb))
        return NET_RX_DROP;

    if (!skb->iif)
        skb->iif = skb->dev->ifindex;

    null_or_orig = NULL;
    orig_dev = skb->dev;
    if (orig_dev->master) {
        if (skb_bond_should_drop(skb))
            null_or_orig = orig_dev; /* deliver only exact match */
        else
            skb->dev = orig_dev->master;
    }

    __get_cpu_var(netdev_rx_stat).total++;

    skb_reset_network_header(skb);
    skb_reset_transport_header(skb);
    skb->mac_len = skb->network_header - skb->mac_header;

    pt_prev = NULL;

    rcu_read_lock();

#ifdef CONFIG_NET_CLS_ACT
    if (skb->tc_verd & TC_NCLS) {
        skb->tc_verd = CLR_TC_NCLS(skb->tc_verd);
        goto ncls;
    }
#endif

    list_for_each_entry_rcu(ptype, &ptype_all, list) {
        if (ptype->dev == null_or_orig || ptype->dev == skb->dev ||
            ptype->dev == orig_dev) {
            if (pt_prev)
                ret = deliver_skb(skb, pt_prev, orig_dev);
            pt_prev = ptype;
        }
    }

#ifdef CONFIG_NET_CLS_ACT
    skb = handle_ing(skb, &pt_prev, &ret, orig_dev);
    if (!skb)
        goto out;
ncls:
#endif

    skb = handle_bridge(skb, &pt_prev, &ret, orig_dev);
    if (!skb)
        goto out;
    skb = handle_macvlan(skb, &pt_prev, &ret, orig_dev);
    if (!skb)
        goto out;

    type = skb->protocol;
    list_for_each_entry_rcu(ptype,
            &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {
        if (ptype->type == type &&
            (ptype->dev == null_or_orig || ptype->dev == skb->dev ||
             ptype->dev == orig_dev)) {
            if (pt_prev)
                ret = deliver_skb(skb, pt_prev, orig_dev);
            pt_prev = ptype;
        }
    }

    if (pt_prev) {
        ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
    } else {
        kfree_skb(skb);
        /* Jamal, now you will not able to escape explaining
         * me how you were going to use this. :-)
         */
        ret = NET_RX_DROP;
    }

out:
    rcu_read_unlock();
    return ret;
}

It looks like the call to kfree_skb will only happen when the skb->dev is not registered in any of the protocol's ptype list, so that the pt_prev remains NULL after the 2 loops against the ptype lists. This does not make sense as the system is dropping only a small part of all packages - meaning the device is "most of time registered in protocol ptype list but sometime not there".

So, question is - what mistakes did I make in understanding the dropwatch result and the netif_receive_skb code? And what is a more reasonable explanation of the packet drops being reported at this function?

1
  • What does ethtool -S eth0 | grep -vw 0 say (replace eth0 with your interface)? Apr 30, 2020 at 8:29

1 Answer 1

1

Maybe this excerpt from https://access.redhat.com/solutions/657483 helps explain the reasons for dropping there:

The RT and RHEL7 kernels contains code that updates the rx_dropped counter for other non-error conditions.

  • The softnet backlog full
  • Bad VLAN tags
  • Packets received with unknown or unregistered protocols
  • IPv6 frames when the server is configured only for ipv4

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .