0

I am trying to debug my kernel module. When I run it I get the following kernel warnings, but it seems that there is no informative message like other warnings I've seen. Is it possible to get any useful info out of this?


Some more info:

The module is called firewall, it diverts tcp packets to a proxy server in user space, and the proxy then sends the tcp data it receives to the intended destination.

This happens when processing an http response by simply receiving all the data on one socket and calling sendall on another. The warning doesn't happen when all the response comes in one packet, but does when the http payload data is segmented into several tcp packets.

The proxy is written in python. It seems strange to me that in the warning it says "python tainted". Can userspace applications cause kernel warnings?


I tried only receiving a large file in the proxy but not sending it and did not get any errors, and the system didn't freeze at any point. The problem happens only on calling socket.sendall/socket.send

reducing the read buffer size and then sending smaller chunks causes the system to lockup faster.

Turning off both gso, tso with ethtool prevents the error messages, but the system still locks up after the same amount of time, making me wonder if the warnings are even tied to the lockup

[16795.153478] ------------[ cut here ]------------
[16795.153489] WARNING: at /build/buildd/linux-3.2.0/net/core/dev.c:1970 skb_gso_segment+0x2e9/0x360()
[16795.153492] Hardware name: VirtualBox
[16795.153495] e1000: caps=(0x40014b89, 0x401b4b89) len=2948 data_len=0 ip_summed=0
[16795.153497] Modules linked in: firewall(O) vesafb vboxsf(O) snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device joydev psmouse snd soundcore serio_raw i2c_piix4 snd_page_alloc vboxguest(O) video bnep mac_hid rfcomm bluetooth parport_pc ppdev lp parport usbhid hid e1000 [last unloaded: firewall]
[16795.153529] Pid: 7644, comm: python Tainted: G        W  O 3.2.0-37-generic-pae #58-Ubuntu
[16795.153532] Call Trace:
[16795.153540]  [<c105a822>] warn_slowpath_common+0x72/0xa0
[16795.153544]  [<c14ad2b9>] ? skb_gso_segment+0x2e9/0x360
[16795.153548]  [<c14ad2b9>] ? skb_gso_segment+0x2e9/0x360
[16795.153551]  [<c105a8f3>] warn_slowpath_fmt+0x33/0x40
[16795.153555]  [<c14ad2b9>] skb_gso_segment+0x2e9/0x360
[16795.153561]  [<c14b01ce>] dev_hard_start_xmit+0xae/0x4c0
[16795.153568]  [<f9a6f2fd>] ? divertPacket+0x7d/0xe0 [firewall]
[16795.153574]  [<c14c8151>] sch_direct_xmit+0xb1/0x180
[16795.153578]  [<f9a6f941>] ? hook_localout+0x71/0xe0 [firewall]
[16795.153582]  [<c14b06d6>] dev_queue_xmit+0xf6/0x370
[16795.153586]  [<c14c6459>] ? eth_header+0x29/0xc0
[16795.153590]  [<c14b73f0>] neigh_resolve_output+0x100/0x1c0
[16795.153594]  [<c14c6430>] ? eth_rebuild_header+0x80/0x80
[16795.153598]  [<c14dec62>] ip_finish_output+0x152/0x2e0
[16795.153602]  [<c14df75f>] ip_output+0xaf/0xc0
[16795.153605]  [<c14dd340>] ? ip_forward_options+0x1d0/0x1d0
[16795.153609]  [<c14deec0>] ip_local_out+0x20/0x30
[16795.153612]  [<c14defee>] ip_queue_xmit+0x11e/0x3c0
[16795.153617]  [<c10841c5>] ? getnstimeofday+0x55/0x120
[16795.153622]  [<c14f4de7>] tcp_transmit_skb+0x2d7/0x4a0
[16795.153626]  [<c14f5786>] tcp_write_xmit+0x146/0x3a0
[16795.153630]  [<c14f5a4c>] __tcp_push_pending_frames+0x2c/0x90
[16795.153634]  [<c14e7d2b>] tcp_sendmsg+0x71b/0xae0
[16795.153638]  [<c104a33d>] ? update_curr+0x1ed/0x360
[16795.153642]  [<c1509c23>] ? inet_recvmsg+0x73/0x90
[16795.153646]  [<c1509ca0>] inet_sendmsg+0x60/0xa0
[16795.153650]  [<c149ae27>] sock_sendmsg+0xf7/0x120
[16795.153655]  [<c1044648>] ? ttwu_do_wakeup+0x28/0x130
[16795.153660]  [<c1036a98>] ? default_spin_lock_flags+0x8/0x10
[16795.153667]  [<c149ce7e>] sys_sendto+0x10e/0x150
[16795.153672]  [<c1117e7f>] ? handle_pte_fault+0x28f/0x2c0
[16795.153675]  [<c111809e>] ? handle_mm_fault+0x15e/0x2c0
[16795.153679]  [<c15ab9c7>] ? do_page_fault+0x227/0x490
[16795.153681]  [<c149cefb>] sys_send+0x3b/0x40
[16795.153684]  [<c149d842>] sys_socketcall+0x162/0x2c0
[16795.153687]  [<c15af55f>] sysenter_do_call+0x12/0x28
[16795.153689] ---[ end trace 3170256120cbbc8f ]---
1

Have you tried following backwards from end of stack trace with addr2line?

For example looking at the last line sysenter_do_call+0x12/0x28

It tells us that the offset is 0x12 and the length is 0x28

$ addr2line -e [path-to-kernel-module-with-issue] 0xc15af55f

and so on...gdb is another alternative in terms of breaking down the stack trace into lines.

However, I am not completely sure how you are arriving at a kernel-panic, as all I am seeing in the log excerpt you provided is a warning. Does it result in a crash/kernel-panic message after the stack-trace you posted?

-------as far as the stack trace posted: it has to do with the general segmentation offload and the skbuffer not being happy with the ip_summed checksum, disabling large\general receiver offload with

$ethtool -k [NIC] lro off $ethtool -k [NIC] gro off

might be a possible workaround. Also, skipping checksum check with skb->ip_summed = CHECKSUM_UNNECESSARY might also solve this issue, depending on the purpose of the setup.

  • Sorry, the tag is misleading, it doesn't cause a kernel panic in dmesg but if I run the program enough times or on a large enough file (try to transfer a large enough file) the system locks up. It seems that there is a warning for every packet with http data. Is it possible that the kernel runs out of memory because my program floods dmesg? – Eloo Feb 24 at 20:33
  • No! After reexamining the stacktrace more closely, what causes this is warning if I am not mistaken is your general segmentation offload(gso) sk buffer is not satisfied with the ip_summed checksum and will kick off this warning. You should try turning off large receiver offload or generic receiver offload with ethtool -k [your NIC device, ethX etc.] lro off or ethtool -k gro off . – BarBar1234 Feb 24 at 21:18
  • I edit the skb of packets that enter/leave my firewall and then fix their checksum in divert_packet. Is it possible that for segmented data something gets messed up there? – Eloo Feb 25 at 8:56
  • I tried only receiving a large file in the proxy but not sending it and did not get any errors. Seems the problem is sending the data. Tried turning off tso but did not help – Eloo Feb 25 at 9:43
  • 1
    Ok so the size has nothing to do with it and what crashed my system was incorrect use of spinlocks. These warnings have nothing to do with the crashes. – Eloo Feb 26 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.