34

I've seen a few bug reports and questions (on stackexchange and elsewhere) regarding a nagging "BUG: soft lockup - CPU#<n> stuck for <dt>s!". So far, I haven't found any clue as to what to do or try (rather, the clues I've found and followed haven't stopped this from happening). I am further concerned about this because:

  1. the frequency of these events seems to have been slowly on the rise lately (over 700 per month),
  2. yum update and reboot slowed it down a bit for a while but I have seen some lockups starting to happen again,
  3. several processes (if not the whole host, it's hard to tell), certainly including all my interactive shells are frozen for some amount of time when it happens,
  4. I'm not sure whether it is related, but I see lots of log/messages related to ntpd not being able to update the clock.

The following is an excerpt of $(grep 'soft lockup' /var/log/messages*):

Mar 22 10:02:35 localhost kernel: BUG: soft lockup - CPU#15 stuck for 10s! [kjournald:1048]
Mar 22 10:02:36 localhost kernel: BUG: soft lockup - CPU#0 stuck for 10s! [postgres:5372]
Mar 22 10:02:36 localhost kernel: BUG: soft lockup - CPU#8 stuck for 10s! [postgres:5368]
Mar 22 10:02:37 localhost kernel: BUG: soft lockup - CPU#0 stuck for 10s! [postgres:5372]
Mar 22 10:02:37 localhost kernel: BUG: soft lockup - CPU#8 stuck for 10s! [postgres:5368]
Mar 22 10:02:38 localhost kernel: BUG: soft lockup - CPU#0 stuck for 10s! [postgres:5372]
Mar 22 10:02:38 localhost kernel: BUG: soft lockup - CPU#8 stuck for 10s! [postgres:5368]
Mar 22 10:02:39 localhost kernel: BUG: soft lockup - CPU#0 stuck for 10s! [postgres:5372]
Mar 22 10:02:39 localhost kernel: BUG: soft lockup - CPU#8 stuck for 10s! [postgres:5368]
Mar 22 10:02:40 localhost kernel: BUG: soft lockup - CPU#15 stuck for 25s! [swapper:0]
Mar 22 15:42:16 localhost kernel: BUG: soft lockup - CPU#8 stuck for 25s! [kjournald:1048]
Mar 22 18:22:13 localhost kernel: BUG: soft lockup - CPU#15 stuck for 10s! [postgres:21356]
Mar 22 18:22:20 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [java:8653]
Mar 22 18:22:20 localhost kernel: BUG: soft lockup - CPU#8 stuck for 72s! [kjournald:1048]
Mar 22 21:21:37 localhost kernel: BUG: soft lockup - CPU#12 stuck for 29s! [kjournald:1048]
Mar 22 21:22:07 localhost kernel: BUG: soft lockup - CPU#12 stuck for 27s! [kjournald:1048]
Mar 23 02:01:47 localhost kernel: BUG: soft lockup - CPU#8 stuck for 10s! [kblockd/8:276]
Mar 23 02:02:22 localhost kernel: BUG: soft lockup - CPU#8 stuck for 34s! [kblockd/8:276]

This happens to random processes, and seems fairly well distributed over the 16 "cores" of that virtual host.

The host is an AWS EC2 "cc1.4xlarge" instance, with an AMI named "EC2 CentOS 5.5 GPU HVM AMI (Driver 260.19.29) (ami-42a2532b)". It seems to be virtualized with Xen.

cat /etc/redhat-release yields CentOS release 5.9 (Final). 'free' reports 21G of RAM.

The head of dmesg is:

Linux version 2.6.18-348.3.1.el5 ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)) #1 SMP Mon Mar 11 19:39:25 EDT 2013
Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet console=tty0 console=ttyS0,115200n8
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000010000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000c0000000 (usable)
 BIOS-e820: 00000000fc000000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 00000005dd800000 (usable)
DMI 2.4 present.
DMI: Xen HVM domU, BIOS 3.4.3-2.6.18 08/29/2012
ACPI: RSDP (v002    Xen                                ) @ 0x00000000000ea020
ACPI: XSDT (v001    Xen      HVM 0x00000000 HVML 0x00000000) @ 0x00000000fc0062b0
ACPI: FADT (v004    Xen      HVM 0x00000000 HVML 0x00000000) @ 0x00000000fc005ee0
ACPI: MADT (v002    Xen      HVM 0x00000000 HVML 0x00000000) @ 0x00000000fc005fe0
ACPI: SRAT (v001    Xen      HVM 0x00000000 HVML 0x00000000) @ 0x00000000fc0060c0
ACPI: SLIT (v001    Xen      HVM 0x00000000 HVML 0x00000000) @ 0x00000000fc006240
ACPI: HPET (v001    Xen      HVM 0x00000000 HVML 0x00000000) @ 0x00000000fc006270
ACPI: DSDT (v002    Xen      HVM 0x00000000 INTL 0x20090220) @ 0x(null)

The following shows a cumulative count of these "soft lockups" over recent time (the redline is when I did the last yum update followed by reboot): cumul count of soft lockups.

The following shows the histogram of duration (how long is the host stuck): duration histogram.

Improve this question
2
  • 1
    Tons of possible reasons. I had it once in a KVM instance. The cause was the hosts network driver (realtek), which would do something on high network loads the virtualization did not expect, and voila you get stuck CPU in the VMs. So basically a bug in the network driver which triggered other bugs further down the road. Solution was to switch to a different kernel version (on the host) which did not trigger that particular behaviour.
    – frostschutz
    Mar 28, 2013 at 1:37
  • 1
    We got this error message, because some VMs had more vcpus configured than were physical CPUs in the new server, we moved our Xen host to.
    – Jörg Ludwig
    Aug 9, 2014 at 9:48

2 Answers 2

Reset to default
11

I also have this issue on Xen 4.2 with 3.6 and 3.8 Kernel (AlpineLinux).

I googled around and by adding clocksource=jiffies to my kernel i fixed it. Instead of jiffies you could also try "pit".

There are also reports of disabling C-states in the BIOS.

Improve this answer
4
  • 4
    What do those kernel parameters do?
    – Burhan Ali
    Apr 1, 2013 at 20:02
  • 2
    Clocksource seems pretty obvious to me and c-states is power states of the CPU.
    – Franz Bettag
    Apr 4, 2013 at 15:43
  • +1. Disabling c-states worked for me.
    – Andrew Ensley
    Jul 28, 2013 at 22:26
  • I would not recommend using pit as the PIT is pretty old and is not as precise as the other timers.
    – xdevs23
    Jun 9 at 10:58
2

I had same problem with my Thinkpad T520. But instead of hacking away at kernel I did something more simple. First off I am using Centos7 I installed base system all worked fine. I then added GNOME GUI later which is when I started getting the problems mentioned above. I notice that a lot of manufacturers set up for Windows installs. The graphics card is setup usually for Win7(NVIDIA OPTIMUS) I reset it to integrated graphics mode and no more hanging/errors. How to do it? Reboot your Thinkpad hit F1 or blue thinkvantage button to get into BIOS. Go to graphics select integrated graphics then F10 to save and exit. There are 3 setting for this card: Integrated,Discrete and NVIDIA OPTIMUS(Win7 only?) Hope this saves someone some time?

Improve this answer
1
  • Sigh, like mostly everything else, installing stuff separately is a no-no. Back to the bloated desktop version with Office and other crap :(
    – killjoy
    May 29, 2018 at 18:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .