1

So I have a Centos 6.6 box running as a LAMP stack.

PHP 5.3.3 | Apache 2.2.15 | MySQL Server 5.1.73

This box is a VM running on XenServer 6.5, It has 4Gb memory available.

             total       used       free     shared    buffers     cached
 Mem:          3881        179       3701          0          3         31
 -/+ buffers/cache:        144       3736
 Swap:         3583         37       3546

The box hosts one website, setup as a virtual host within Apache. The site itself is a low traffic WordPress site.

I'm constantly seeing a 'out of memory: kill process (xxxx) httpd' error. Of course, at this point the website becomes unresponsive, and often the server takes quite a while to listen to my commands.

Even if I reboot the box and bring httpd online, within about 5 mins or so, I can see the OOM warning and apache is killed off.

I'm not the developer of the site, I'm just unfortunate enough to be hosting it and I'm going crazy trying to keep this bloody server online.

Any help at all is appriciated!

output of dmesg:

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-573.26.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed May 4 00:57:44 UTC 2016
Command line: ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD console=hvc0  KEYTABLE=us rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root rd_NO_DM rhgb quiet graphical utf8
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000100000000 (usable)
DMI not present or invalid.
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
No AGP bridge found
last_pfn = 0x100000 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-0000000100000000
 0000000000 - 0100000000 page 4k
kernel direct mapping tables up to 100000000 @ 100000-905000
RAMDISK: 0205d000 - 061ad000
ACPI: Deleted _OSI()
ACPI: Deleted _OSI()
No NUMA configuration found
Faking a node at 0000000000000000-0000000100000000
Bootmem setup node 0 0000000000000000-0000000100000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [000000000003c000 -  000000000005bfff] pages 20
(7 early reservations) ==> bootmem [0000000000 - 0100000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [00069b0000 - 00069e9000]   XEN PAGETABLES ==> [00069b0000 - 00069e9000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 000203cb64]    TEXT DATA BSS ==> [0001000000 - 000203cb64]
  #4 [000205d000 - 00061ad000]          RAMDISK ==> [000205d000 - 00061ad000]
  #5 [00061ad000 - 00069b0000]   XEN START INFO ==> [00061ad000 - 00069b0000]
  #6 [0000100000 - 00008c9000]          PGTABLE ==> [0000100000 - 00008c9000]
Reserving 129MB of memory at 112MB for crashkernel (System RAM: 4096MB)
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00100000
On node 0 totalpages: 1048479
  DMA zone: 56 pages used for memmap
  DMA zone: 1996 pages reserved
  DMA zone: 1947 pages, LIFO batch:0
  DMA32 zone: 14280 pages used for memmap
  DMA32 zone: 1030200 pages, LIFO batch:31
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 2 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
nr_irqs_gsi: 16
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 100100000 (gap: 100100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 4.1.5 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Embedded 30 pages/cpu @ffff880006a1f000 s91224 r8192 d23464 u122880
pcpu-alloc: s91224 r8192 d23464 u122880 alloc=30*4096
pcpu-alloc: [0] 0 [0] 1
trying to map vcpu_info 0 at ffff880006a29020, mfn 1354fe, offset 32
cpu 0 using vcpu_info at ffff880006a29020
trying to map vcpu_info 1 at ffff880006a47020, mfn 1354e0, offset 32
cpu 1 using vcpu_info at ffff880006a47020
Xen: using vcpu_info placement
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 1032147
Policy zone: DMA32
Kernel command line: ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD console=hvc0  KEYTABLE=us rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M rd_LVM_LV=VolGroup/lv_root rd_NO_DM rhgb quiet graphical utf8
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
Memory: 3903712k/4194304k available (5398k kernel code, 388k absent, 290204k reserved, 7010k data, 1296k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:288
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 16777216 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
  alloc irq_desc for 287 on node 0
  alloc kstat_irqs on node 0
Detected 1995.021 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.04 BogoMIPS (lpj=1995021)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: Unsupported number of siblings 4
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 22136 entries in 87 pages
  alloc irq_desc for 286 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 285 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 284 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 283 on node 0
  alloc kstat_irqs on node 0
Performance Events: unsupported p6 CPU model 23 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
installing Xen timer for CPU 1
  alloc irq_desc for 282 on node 0
  alloc kstat_irqs on node 0
SMP alternatives: switching to SMP code
  alloc irq_desc for 281 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 280 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 279 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 278 on node 0
  alloc kstat_irqs on node 0
CPU: Unsupported number of siblings 4
Brought up 2 CPUs
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2672 bytes
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
  alloc irq_desc for 277 on node 0
  alloc kstat_irqs on node 0
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
last_pfn = 0x100000 max_arch_pfn = 0x400000000
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 66880k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
futex hash table entries: 512 (order: 3, 32768 bytes)
audit: initializing netlink socket (disabled)
type=2000 audit(1463989007.461:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 7755
SELinux:  Registering netfilter hooks
ksign: Installing public key data
Loading keyring
- Added public key AC9C0BB1152BECC6
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
  alloc irq_desc for 276 on node 0
  alloc kstat_irqs on node 0
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
GRE over IPv4 demultiplexor driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vbd/51760
XENBUS: Device with no driver: device/vif/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1296k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 728k freed
Freeing unused kernel memory: 1572k freed
dracut: dracut-004-388.el6
dracut: rd_NO_LUKS: removing cryptoluks activation
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.29.0-ioctl (2014-10-28) initialised: dm-devel@redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
dracut: rd_NO_MD: removing MD RAID activation
xlblk_init: register_blkdev major: 202
  alloc irq_desc for 275 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 274 on node 0
  alloc kstat_irqs on node 0
blkfront: xvda: barrier: enabled; persistent grants: disabled; indirect descriptors: disabled;
 xvda: xvda1 xvda2
blkfront: xvdd: barrier: enabled; persistent grants: disabled; indirect descriptors: disabled;
 xvdd: unknown partition table
Changing capacity of (202, 0) to 73400320 sectors
xvda: detected capacity change from 0 to 37580963840
Changing capacity of (202, 48) to 232808 sectors
xvdd: detected capacity change from 0 to 119197696
dracut: Scanning devices xvda2  for LVM logical volumes VolGroup/lv_swap VolGroup/lv_root
dracut: inactive '/dev/VolGroup/lv_root' [31.01 GiB] inherit
dracut: inactive '/dev/VolGroup/lv_swap' [3.50 GiB] inherit
EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts:
dracut: Mounted root filesystem /dev/mapper/VolGroup-lv_root
dracut: Loading SELinux policy
SELinux: 2048 avtab hash slots, 305202 rules.
SELinux: 2048 avtab hash slots, 305202 rules.
SELinux:  9 users, 12 roles, 4199 types, 234 bools, 1 sens, 1024 cats
SELinux:  81 classes, 305202 rules
SELinux:  Completing initialization.
SELinux:  Setting up existing superblocks.
SELinux: initialized (dev dm-0, type ext4), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses transition SIDs
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev devtmpfs, type devtmpfs), uses transition SIDs
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
type=1403 audit(1463989009.276:2): policy loaded auid=4294967295 ses=4294967295
dracut:
dracut: Switching root
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
udev: starting version 147
Initialising Xen virtual ethernet driver.
  alloc irq_desc for 273 on node 0
  alloc kstat_irqs on node 0
EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts:
SELinux: initialized (dev xvda1, type ext4), uses xattr
Adding 3670012k swap on /dev/mapper/VolGroup-lv_swap.  Priority:-1 extents:1 across:3670012k SS
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
type=1305 audit(1463989021.731:3): audit_pid=919 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
SELinux: initialized (dev xenfs, type xenfs), uses genfs_contexts
eth0: no IPv6 routers present
  • 1
    could you please post dmesg output ? – Rahul May 23 '16 at 7:39
  • was too long to add as comment, so updated original post with the output – John May 23 '16 at 7:55
  • Where do you see the OOM messages? Either the relevant Errors from /var/log/messages should also be added and formatted as code or the relevant lines from /var/log/httpd/error_log. – ILMostro_7 May 23 '16 at 7:59
  • straight on the terminal... Out of memory: Kill process 1872 (httpd) score 8 or sacrifice child Killed process 1872, UID 48, (httpd) total-vm:340164kB, anon-rss:19252kB, file-rss:688kB Out of memory: Kill process 1841 (httpd) score 8 or sacrifice child Killed process 1841, UID 48, (httpd) total-vm:333500kB, anon-rss:9796kB, file-rss:464kB Out of memory: Kill process 1873 (httpd) score 8 or sacrifice child Killed process 1873, UID 48, (httpd) total-vm:338648kB, anon-rss:16408kB, file-rss:592kB etc etc... – John May 23 '16 at 8:01
  • 1
    you should always add requested output of commands or file contents etc to your question, not as comments. in comments they're unformatted and often-unreadable gibberish. in the question, they can be formatted with the {} icon in the editor. – cas May 23 '16 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.