0

I have a server crashing almost everyday. It runs on Debian, kernel 4.9.0-8-amd64.

Here are the logs written before the latest crash:

Aug 14 02:21:24 server6 kernel: INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 14 02:21:24 server6 kernel:         14-...: (1 GPs behind) idle=d63/140000000000000/0 softirq=28564730/28564730 fqs=2304
Aug 14 02:21:24 server6 kernel:         15-...: (2 GPs behind) idle=77b/140000000000000/0 softirq=28687922/28687922 fqs=2304
Aug 14 02:21:24 server6 kernel:         (detected by 8, t=5252 jiffies, g=7706995, c=7706994, q=30757)
Aug 14 02:21:24 server6 kernel: Task dump for CPU 14:
Aug 14 02:21:24 server6 kernel: aws             R  running task        0 18987  18979 0x00000008
Aug 14 02:21:24 server6 kernel:  ffffffff8978cce6 0000000000000287 024000c01f37ccc0 ffff8b7ec409a000
Aug 14 02:21:24 server6 kernel:  ffff8b821f37cc00 0000000300000001 0000000501000000 ffffa731db1e7dc0
Aug 14 02:21:24 server6 kernel:  00000000024280ca ffffffff8a4e9da0 0000000000000000 ffff8b7eb1025bb8
Aug 14 02:21:24 server6 kernel: Call Trace:
Aug 14 02:21:24 server6 kernel:  [<ffffffff8978cce6>] ? __alloc_pages_nodemask+0xf6/0x260
Aug 14 02:21:24 server6 kernel:  [<ffffffff89801ad8>] ? mem_cgroup_commit_charge+0x78/0x4b0
Aug 14 02:21:24 server6 kernel:  [<ffffffff897badc8>] ? handle_mm_fault+0xcd8/0x1310
Aug 14 02:21:24 server6 kernel:  [<ffffffff89662318>] ? __do_page_fault+0x278/0x4f0
Aug 14 02:21:24 server6 kernel:  [<ffffffff89c19b79>] ? retint_user+0x8/0x2c
Aug 14 02:21:24 server6 kernel: Task dump for CPU 15:
Aug 14 02:21:24 server6 kernel: aws             R  running task        0 19006  18979 0x00000008
Aug 14 02:21:24 server6 kernel:  ffffffff8978cce6 0000000000000287 024000c01f37ccc0 ffff8b7edca4e000
Aug 14 02:21:24 server6 kernel:  ffff8b821f37cc00 0000000300000001 0000000501000000 ffffa731db23fdc0
Aug 14 02:21:24 server6 kernel:  00000000024280ca ffffffff8a4e9da0 0000000000000000 ffff8b80de6e1190
Aug 14 02:21:24 server6 kernel: Call Trace:
Aug 14 02:21:24 server6 kernel:  [<ffffffff8978cce6>] ? __alloc_pages_nodemask+0xf6/0x260
Aug 14 02:21:24 server6 kernel:  [<ffffffff89801ad8>] ? mem_cgroup_commit_charge+0x78/0x4b0
Aug 14 02:21:24 server6 kernel:  [<ffffffff897badc8>] ? handle_mm_fault+0xcd8/0x1310
Aug 14 02:21:24 server6 kernel:  [<ffffffff89662318>] ? __do_page_fault+0x278/0x4f0
Aug 14 02:21:24 server6 kernel:  [<ffffffff89c19b79>] ? retint_user+0x8/0x2c
Aug 14 02:21:27 server6 kernel: NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [node_exporter:2317]
Aug 14 02:21:27 server6 kernel: Modules linked in: nvidia_uvm(O) veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrac
Aug 14 02:21:27 server6 kernel:  shpchp serio_raw sp5100_tco evdev acpi_cpufreq irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_in
Aug 14 02:21:27 server6 kernel: CPU: 9 PID: 2317 Comm: node_exporter Tainted: P        W  O    4.9.0-8-amd64 #1 Debian 4.9.144-3.1
Aug 14 02:21:27 server6 kernel: Hardware name: System manufacturer System Product Name/ROG STRIX X470-F GAMING, BIOS 4024 09/07/2018
Aug 14 02:21:27 server6 kernel: task: ffff8b8202bfb100 task.stack: ffffa731c35d4000
Aug 14 02:21:27 server6 kernel: RIP: 0010:[<ffffffff896fe5b3>]  [<ffffffff896fe5b3>] smp_call_function_single+0xd3/0x130
Aug 14 02:21:27 server6 kernel: RSP: 0018:ffffa731c35d7d18  EFLAGS: 00000202
Aug 14 02:21:27 server6 kernel: RAX: 0000000000000000 RBX: 000000000000000f RCX: ffffc731bfdc3500
Aug 14 02:21:27 server6 kernel: RDX: 0000000000000001 RSI: ffffa731c35d7d28 RDI: ffffa731c35d7d28
Aug 14 02:21:27 server6 kernel: RBP: ffffffffc02ab0a0 R08: ffffffffffff8000 R09: 000000000000ffff
Aug 14 02:21:27 server6 kernel: R10: 000000000000000d R11: ffff8b810964a538 R12: ffffffffc02ab0a0
Aug 14 02:21:27 server6 kernel: R13: ffffa731c35d7d98 R14: 0000000000000001 R15: ffff8b8030158840
Aug 14 02:21:27 server6 kernel: FS:  000000c000302090(0000) GS:ffff8b821ee40000(0000) knlGS:0000000000000000
Aug 14 02:21:27 server6 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 14 02:21:27 server6 kernel: CR2: 00007f9cf8b25000 CR3: 0000000402d49000 CR4: 00000000003406f0
Aug 14 02:21:27 server6 kernel: Stack:
Aug 14 02:21:27 server6 kernel:  000000410002e000 ffffa731c35d7d90 ffffc731bfdc3500 ffffffffc02ab0a0
Aug 14 02:21:27 server6 kernel:  ffffa731c35d7d98 0000000000000003 cca4302174496634 ffffffff89e11740
Aug 14 02:21:27 server6 kernel:  ffffffff8a4afd80 ffffffff896fe6b6 ffffffff89e11740 ffff8b8204cb6800
Aug 14 02:21:27 server6 kernel: Call Trace:
Aug 14 02:21:27 server6 kernel:  [<ffffffffc02ab0a0>] ? cpu_freq_write_amd+0x20/0x20 [acpi_cpufreq]
Aug 14 02:21:27 server6 kernel:  [<ffffffff896fe6b6>] ? smp_call_function_any+0x36/0xc0
Aug 14 02:21:27 server6 kernel:  [<ffffffffc02ab21a>] ? get_cur_val+0x7a/0xf0 [acpi_cpufreq]
Aug 14 02:21:27 server6 kernel:  [<ffffffffc02ab060>] ? cpu_freq_write_intel+0x40/0x40 [acpi_cpufreq]
Aug 14 02:21:27 server6 kernel:  [<ffffffffc02ab97d>] ? get_cur_freq_on_cpu+0x8d/0x110 [acpi_cpufreq]
Aug 14 02:21:27 server6 kernel:  [<ffffffff896a5674>] ? try_to_wake_up+0x54/0x3c0
Aug 14 02:21:27 server6 kernel:  [<ffffffff89ad3dd5>] ? __cpufreq_get+0x25/0xa0
Aug 14 02:21:27 server6 kernel:  [<ffffffff89ad4efe>] ? show_cpuinfo_cur_freq+0xe/0x50
Aug 14 02:21:27 server6 kernel:  [<ffffffff89ad3d93>] ? show+0x33/0x50
Aug 14 02:21:27 server6 kernel:  [<ffffffff8988de28>] ? sysfs_kf_seq_show+0xb8/0x130
Aug 14 02:21:27 server6 kernel:  [<ffffffff89831636>] ? seq_read+0x106/0x400
Aug 14 02:21:27 server6 kernel:  [<ffffffff8980b261>] ? vfs_read+0x91/0x130
Aug 14 02:21:27 server6 kernel:  [<ffffffff8980c732>] ? SyS_read+0x52/0xc0
Aug 14 02:21:27 server6 kernel:  [<ffffffff89603b7d>] ? do_syscall_64+0x8d/0xf0
Aug 14 02:21:27 server6 kernel:  [<ffffffff89c1924e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
Aug 14 02:21:27 server6 kernel: Code: 25 28 00 00 00 75 73 48 83 c4 38 5b 5d c3 48 8d 74 24 10 48 89 d1 89 df 48 89 ea e8 28 fe ff ff 8b
Aug 14 02:21:27 server6 kernel: NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [aws:18986]

Can someone help me understand what's the issue?

Edit:

For the first message, it seems like it's related to the aws process. I have the aws CLI (which is multithreaded) configured with max_concurrent_requests = 100 (instead of 10 by default). I'm decreasing it to see if that resolves the issue.

New contributor
GG. is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

GG. is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.