3

I have an m1.small EC2 instance that is mostly just running Apache as a web server for several simple PHP web sites (that use RDS as a database). The server is constantly with a very high load average - around 8, and never below 5. This causes my web sites to be annoyingly slow, much more then I'd expect from a supposedly 2.6GHz CPU with 1.7GB RAM.

top and friends shows that other then ~50% "steal" time (AFAIU, time stolen by the hypervisor for other VMs on the same CPU), the rest is almost entirely "IRQ" time.

mpstat says that this time is spend on a very stable 191.59 interrupts/sec (i.e. that number almost doesn't change over time), and according to '/proc/interrupts' these are mostly spend on xen-percpu-virq timer0 and xen-dyn-event eth0.

What are these, and how can I get the EC2 instance to lower the load and spend some more time on my PHP sites?

Improve this question
9
  • What does output of /cat /proc/interrupts show?
    – slm ♦
    Jun 7, 2014 at 17:32
  • I'm a little suspicious that you're expecting too much from the m1.small.EC2 instances. blog.patrickmeenan.com/2012/05/ec2-performance.html. Also this: blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff
    – slm ♦
    Jun 7, 2014 at 18:20
  • /proc/interrupts just shows high counts for the items specified in the question, and low or zero for everything else. Regarding performance - I don't think I'm expected a lot - a celeron class CPU should be able to run a requested through a somewhat customized wordpress site in less then 16 seconds (which is my current performance).
    – Guss
    Jun 8, 2014 at 6:37
  • 1
    This sounded similar to your issue: serverfault.com/questions/446173/…
    – slm ♦
    Jun 8, 2014 at 12:25
  • 1
    Actually, @sim, reading the serverfault question again, and after finding an AWS support thread with a similar issue, I think this is relevant. Thanks!
    – Guss
    Jun 9, 2014 at 5:56

1 Answer 1

Reset to default
1

After looking at it for a while, this seems like a "hardware issue" - most likely the physical server hosting the VM is under heavy load from other VMs, and the m1 type machines do not get guaranteed performance, so you get whatever CPU time is left on the host. Worse, if your VM is running on CPU core 0, you also get all the interrupt handling on the machine. See this Xen wiki article under "HVM VM's first (and possibly only) VCPU is fully utilised.". Also see this AWS support thread for other people with a similar interrupt problem.

The only solution at this point is to get your VM migrated to another physical host. This can sometimes be done very simply by stopping the VMs, waiting a few minutes and starting again - if EC2 has assigned another VM to your old host in the mean time, then you can get a different slot and the problem may be solved.

If that doesn't work, the best bet is to create a new image from your VM, and start it on another availability zone. This will cause IPs to be replaced and may require updating security groups and firewall rules. Also make sure you stop your VM before you create the image - to make sure that after the image is created and before the new VM is started, your old VM will not generate new data.

Improve this answer
1
  • 2
    Please note that this answer is very old. Currently stopping a VM will always cause its IP to be lost and a new IP will need to be generated when it is started. Also the underlying technology has also changed quite a lot and this problem is unlikely to happen.
    – Guss
    Dec 10, 2015 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.