Is there any possibility of tracing a packet through all networking layers inside the Linux kernel ? I'm aware of SystemTap, but I'm looking to see if there's another (more high level) solution.

The problem I'm facing is the following: I have a machine acting as a router, with multiple interfaces, a bridge, a firewall with multiple rules and some advanced routing using custom tables. Some incoming packet replies don't get delivered to a local process (even though I see them with tcpdump) and it's difficult to figure out if they are lost because of routing, firewall, or other cause.

So I'm looking for something to help with this, a tool which can "trace packets coming on interface X, from a.b.c.d" and will produce an output of how the packet traveled through the network layers and where it was dropped or delivered or left the system. Like a strace for packets.

Anything like this exists?

EDIT: added more details for clarity.

  • I think that Dtrace May help you . thanks – Hamza Jabbour Jan 20 '17 at 14:25

You can get sometimes information about the packet traversal in the Linux Kernel stack by probing the statistics/SNMP counters exported by the kernel to userspace. For example, by "netstat -s" (or also by "cat /proc/net/snmp", which is a bit less easy to read). So for example, if there was some error due to wrong IPv4 header, then InHdrErrors will be incremented, and "cat /proc/net/snmp" will show that this error occurred; look in the fourth column which you get when you run "cat /proc/net/snmp" (which is called "InHdrErrors") Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors

If you look in the kernel code, you will see that in the ip_recv() method, which is the handler for IPv4 traffic, there is a check for minimum header length, or that the version member
in the IPv4 header is "4", and if it is not so, the IPSTATS_MIB_INHDRERRORS counter is incremented:

See: http://lxr.free-electrons.com/source/net/ipv4/ip_input.c#L444 http://lxr.free-electrons.com/source/net/ipv4/ip_input.c#L494 http://lxr.free-electrons.com/source/include/uapi/linux/snmp.h

And of course there are other types of errors, and also there are SNMP counters/stats for normal traversal of a packet in the Linux Kernel stack, which you can explore when running "netstat -s" and/or "cat /proc/net/snmp".

Rami Rosen

| improve this answer | |
  • True, but this is unreliable guesswork, as there's no possibility of following a certain packet through statistics (if the machine has traffic, the counters will be bumped up with no way which packet caused the increment). – Unknown Jan 25 '17 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.