Why would the kernel drop packets?

Question

I interrupted tcpdump with Ctrl+C and got this total summary:

579204 packets captured
579346 packets received by filter
142 packets dropped by kernel

What are the "packets dropped by kernel"? Why does that happen?

Answer 1

From tcpdump(1) (tcpdump's man page):

packets ‘‘dropped by kernel’’ (this is the number of packets that were dropped, due to a lack of buffer space, by the packet capture mechanism in the OS on which tcpdump is running, if the OS reports that information to applications;  if not, it will be reported as 0).

A bit of explanation:

The tcpdump program captures raw packets passing through a network interface.  The packets have to be parsed and filtered according to rules specified by you in the command line, and that takes some time, so incoming packets have to be buffered (queued) for processing.  Sometimes there are too many packets, and so they are saved to a buffer.  But they are saved faster than processed, so eventually the buffer runs out of space, and so the kernel drops all further packets until there is some free space in the buffer.

You can increase the buffer size with the -B (--buffer-size) option like this:

tcpdump -B 4096 ....

or

tcpdump --buffer-size=4096 ...

Note that the size is specified in kibibytes, so the lines above set the buffer size to 4 MiB.

Answer 2

One more thing to consider/try is that tcpdump may be spending a lot of time doing DNS queries to resolve IPs to domain names. If you don't need those, try throwing in the -n (no lookups) flag. e.g.:

tcpdump -n port 80

Answer 3

According to man tcpdump:

packets dropped by kernel (this is the number of packets that were dropped, due to a lack of buffer space, by the packet capture mechanism in the OS on which tcpdump is running, if the OS reports that information to applications; if not, it will be reported as 0).

The kernel puts captured packets in a fixed-size capture buffer. If tcpdump doesn't empty that buffer quickly enough, the kernel will begin overwriting old packets in the buffer and correspondingly incrementing the dropped counter. The value of that counter is what you see as "dropped by kernel".

By the way, you can resize the capture buffer: Pass tcpdump the -B option with a KiB size.

Answer 4

Besides what the man page says, there appears to be some additional reason why packets may be dropped by the kernel. I was experiencing 100% packet drop from tcpdump where the only traffic on the network was one 512B packet of PRBS per second. Clearly the buffer space explanation doesn't make sense here - I think the kernel can handle 0.5kiB/s.

Something that came along with my distro (Ubuntu 14.04) may have been doing some sort of smart filtering at the link layer that didn't like my test packets. My workaround was to create a new network namespace as follows:

sudo -i
ip netns add debug
ip link set dev eth0 netns debug
ip netns exec debug bash
ifconfig eth0 1.2.3.4 up

In the inner netns shell, whatever OS processes that were causing problems before are out of the picture and tcpdump shows me all of the packets I expect to see.

Answer 5

I find useful using the tcpdump -c option. This way you can set the number of packets and then stop and you cannot fill out the buffer.

For instance this one will capture the tcp requests on localhost.

tcpdump -ni lo tcp -c 20

Answer 6

I found that in order to not get "dropped by kernel", I had to use all three things that others above have recommended:

tcpdump -c 10000 -n -B 10240