As a side task I administer a RHEL 7 server used as an internal KVM host. Now it runs eight VMs which, but for one, are idle most of their time, or at least not doing anything at all resource intensive. However, new VMs will be periodically added, some of which will not be so idle.
I know what I'm doing as far as setting up the VMs, configuring them, networking them, all the stuff it takes to make them run at least well enough to do their job.
How do I spot the host itself slowing down/reaching performance bottlenecks under new loads? There is ample RAM and CPU cores for the load now, but with an indeterminate number of future VMs spun up, it's only a matter of time before it starts to buckle under new loads. I don't really know what to watch for, to catch performance problems before they become real problems, besides just keeping an eye on RAM usage and the QEMU/KVM processes in top.
This server is an old Dell T620 with two Xeon E5-2660 CPUs and 96 GB RAM,, The VM files are stored on an XFS-formatted RAID-10 array built on a set of 5400-rpm 3.5" disks (which I realize is not ideal, but I have to work with what I have).