3

We asked our server provider to increase ram (from 4gb to 6gb) on our server. After they increased and restarted server, it started giving these messages:

Message from syslogd@www at Jul  1 14:39:04 ...
 kernel:[   30.426516] Oops: 0002 [#1] SMP 

Message from syslogd@www at Jul  1 14:39:04 ...
 kernel:[   30.426677] last sysfs file: /sys/devices/virtual/net/lo/operstate

Message from syslogd@www at Jul  1 14:39:04 ...
 kernel:[   30.431797] Stack:

Message from syslogd@www at Jul  1 14:39:04 ...
 kernel:[   30.432892] Call Trace:

Message from syslogd@www at Jul  1 14:39:04 ...
 kernel:[   30.433380] Code: 00 00 55 53 49 8b 6c 24 08 48 89 fb 4c 39 e5 48 0f 44 e8 48 85 ed 74 3d 48 83 bf 00 02 00 00 00 75 09 83 3d 4e 3f 27 00 00 74 2a <80> 0c 25 5c 00 00 00 01 ff 45 54 ff 83 54 05 00 00 48 83 bb 00 

Message from syslogd@www at Jul  1 14:39:04 ...
 kernel:[   30.436159] CR2: 000000000000005c

And our ERP (Odoo) stopped working after this. Our provider says it is Odoo fault for not working, not because they increased RAM, but everything was running fine before RAM was increased.

Also don't know if it is related, but on that server postgresql was optimized and kernel shared memory was increased (before ram upgrade) - http://www.postgresql.org/docs/9.1/static/kernel-resources.html

Does it look like ram related problem? To me it looks like that, but our server provider can't seem to be able to fix it..

P.S. server is Linux debian squeeze. Kernel info: 2.6.32-5-amd64. And server is run on VM

Update also syslog shows this (fragment of the log):

Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: unknown payload OID
Jul  1 15:20:56 www snmpd[1288]: Unknown payload OID: fileName
Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: Unknown payload OID
Jul  1 15:20:56 www snmpd[1288]: payload OID: fileErrorMsg
Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: unknown payload OID
Jul  1 15:20:56 www snmpd[1288]: Unknown payload OID: fileErrorMsg
Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: Unknown payload OID
Jul  1 15:20:56 www snmpd[1288]: trigger OID: fileErrorFlag
Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: unknown monitor OID
Jul  1 15:20:56 www snmpd[1288]: payload OID: snmperrErrMessage
Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: unknown payload OID
Jul  1 15:20:56 www snmpd[1288]: Unknown payload OID: snmperrErrMessage
Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: Unknown payload OID
Jul  1 15:20:56 www snmpd[1288]: trigger OID: snmperrErrorFlag
Jul  1 15:20:56 www snmpd[1288]: /etc/snmp/snmpd.conf: line 146: Error: unknown monitor OID
Jul  1 15:20:56 www snmpd[1288]: net-snmp: 33 error(s) in config file(s)

Update2

Also it shows this bug. This might be the root cause, but wonder why it happened and how to fix it.

Jul  1 15:57:50 www kernel: [  866.578614] BUG: unable to handle kernel NULL pointer dereference at 000000000000005c
Jul  1 15:57:50 www kernel: [  866.578788] IP: [<ffffffff8128c528>] tcp_send_fin+0x37/0x1ab
Jul  1 15:57:50 www kernel: [  866.578894] PGD 1bcee6067 PUD 1bd558067 PMD 0 
Jul  1 15:57:50 www kernel: [  866.579027] Oops: 0002 [#15] SMP 
Jul  1 15:57:50 www kernel: [  866.579129] last sysfs file: /sys/devices/virtual/net/lo/operstate
Jul  1 15:57:50 www kernel: [  866.579211] CPU 0 
Jul  1 15:57:50 www kernel: [  866.579281] Modules linked in: iptable_filter iptable_mangle ip_tables x_tables loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr parport_pc parport psmouse serio_raw container shpchp pci_hotplug processor i2c_piix4 button ac i2c_core ext3 jbd mbcache dm_mod vmw_pvscsi vmxnet3 sg sd_mod sr_mod cdrom ata_generic crc_t10dif ata_piix libata floppy mptspi mptscsih mptbase e1000 scsi_transport_spi scsi_mod thermal thermal_sys [last unloaded: scsi_wait_scan]
Jul  1 15:57:50 www kernel: [  866.581127] Pid: 2639, comm: python Tainted: G      D    2.6.32-5-amd64 #1 VMware Virtual Platform
Jul  1 15:57:50 www kernel: [  866.581248] RIP: 0010:[<ffffffff8128c528>]  [<ffffffff8128c528>] tcp_send_fin+0x37/0x1ab
Jul  1 15:57:50 www kernel: [  866.581386] RSP: 0018:ffff8801bc937f08  EFLAGS: 00010286
Jul  1 15:57:50 www kernel: [  866.581463] RAX: 0000000000000000 RBX: ffff8801bcf9d480 RCX: ffff8801bdfac901
Jul  1 15:57:50 www kernel: [  866.581555] RDX: ffff880006e189d8 RSI: 0000000000000004 RDI: ffff8801bcf9d480
Jul  1 15:57:50 www kernel: [  866.581648] RBP: ffff8801bae70000 R08: ffff8801badcd030 R09: 0000000000000002
Jul  1 15:57:50 www kernel: [  866.581740] R10: 0000000000000002 R11: ffff8801bcf9d480 R12: ffff8801bcf9d548
Jul  1 15:57:50 www kernel: [  866.581832] R13: ffff8801bdd6a3c0 R14: 00000000ffffffff R15: 00000000ffffffff
Jul  1 15:57:50 www kernel: [  866.581924] FS:  00007fca17123700(0000) GS:ffff880006e00000(0000) knlGS:0000000000000000
Jul  1 15:57:50 www kernel: [  866.582037] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  1 15:57:50 www kernel: [  866.582124] CR2: 000000000000005c CR3: 00000001bd570000 CR4: 00000000000406f0
Jul  1 15:57:50 www kernel: [  866.582234] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul  1 15:57:50 www kernel: [  866.582329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul  1 15:57:50 www kernel: [  866.582421] Process python (pid: 2639, threadinfo ffff8801bc936000, task ffff8801bd510e20)
Jul  1 15:57:50 www kernel: [  866.582537] Stack:
Jul  1 15:57:50 www kernel: [  866.582588]  ffff8801bcf9d480 0000000000000000 0000000000000002 ffffffff8129b6af
Jul  1 15:57:50 www kernel: [  866.582757] <0> 000000000000002c ffff8801bdd6a3c0 0000000000000001 0000000000000001
Jul  1 15:57:50 www kernel: [  866.583032] <0> 00007fca33f36829 ffffffff812423e7 00007fca171236a8 00007fca246bbc20
Jul  1 15:57:50 www kernel: [  866.583291] Call Trace:
Jul  1 15:57:50 www kernel: [  866.583350]  [<ffffffff8129b6af>] ? inet_shutdown+0x97/0xdd
Jul  1 15:57:50 www kernel: [  866.583431]  [<ffffffff812423e7>] ? sys_shutdown+0x3d/0x5d
Jul  1 15:57:50 www kernel: [  866.583512]  [<ffffffff81010b22>] ? system_call_fastpath+0x16/0x1b
Jul  1 15:57:50 www kernel: [  866.583596] Code: 00 00 55 53 49 8b 6c 24 08 48 89 fb 4c 39 e5 48 0f 44 e8 48 85 ed 74 3d 48 83 bf 00 02 00 00 00 75 09 83 3d 4e 3f 27 00 00 74 2a <80> 0c 25 5c 00 00 00 01 ff 45 54 ff 83 54 05 00 00 48 83 bb 00 
Jul  1 15:57:50 www kernel: [  866.585279] RIP  [<ffffffff8128c528>] tcp_send_fin+0x37/0x1ab
Jul  1 15:57:50 www kernel: [  866.585400]  RSP <ffff8801bc937f08>
Jul  1 15:57:50 www kernel: [  866.585468] CR2: 000000000000005c
Jul  1 15:57:50 www kernel: [  866.585716] ---[ end trace 29537c3dcdc7a93f ]---

1 Answer 1

2

That's a kernel bug, Debian bug #789037 aka upstream bug #99161. It was introduced in a recent kernel update, and you probably saw it after the reboot (to install RAM) because you're now running that kernel.

The fix is already available; you need to install it (and reboot).

(BTW: I saw this on some of our servers after an unplanned reboot due to a circuit breaker blowing... The key part to Google was "tcp_send_fin+0x37".)

2
  • Yeah, our server provider just told me that. Was able to restore now.
    – Andrius
    Jul 1, 2015 at 13:42
  • @Andrius Don't forget to upvote if it answers your question. Jul 1, 2015 at 14:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .