Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am scripting a new program that will connect to a cluster of machines in a VM park, and hopefully check their level of usage. The VM clients are a combination of RHEL and CentOS, and runs on multiple KVM hosts.

I am hoping to be able to find the VMs that are no longer needed, by checking what they are being used for now.

What would be good indicators for system usage, and which commands could help me with rating the system accordingly?

I am thinking that if I combine commands like uptime, "chkconfig --list", last and ps I can determine the rate of usage, but at a very innacturate level.

Hope you could help me with this conundrum.

With system usage I am thinking of if the machine is in use, or if it could be shut down, without it harming any applications or users.

With the examples, I can determine if there are servers that are a lot different than the rest, but I would not be able to state if it is in use or not.

share|improve this question
1  
uptime will give you the load of the host. But you should define what you mean by "usage". –  michas May 7 '13 at 23:02
    
and w will give you the list of logged users. If they may launch applications in a non interactive way, ps -eF which give you the list of running processes could be of some help. –  lgeorget May 7 '13 at 23:21
    
Depending on what services they offer, netstat and/or tcpdump may help. –  schaiba May 7 '13 at 23:25
    
Yes, netstat would probably be smart to use. w and ps -eF are also good indicators, but it is difficult to determine a accurate without knowing what is supposed to be running on the server, but should of cource be used with this intent. –  larhauga May 7 '13 at 23:40
add comment

2 Answers

up vote 0 down vote accepted

Monitoring

This is a pretty wide open question. There are tons of applications that you can use to monitor the load of a server's CPU, network traffic, file I/O, etc.

At the most basic level I would start by monitoring the VMs CPU load. You can monitor this in 2 ways, either within the VMs themselves or externally on the VM Hosts.

Monitoring through the KVM Host

Since a VM at it's most basic level is just an application you can simply monitor each VM on a KVM server by watching their corresponding qemu process.

$ ps -C qemu-kvm -o pid,time,etime,pcpu,args|perl -lane 'printf "%-7s %-12s %-12s %-5s %-5s %-s\n", @F[0..3,9,13]'
PID     TIME         ELAPSED      %CPU        
3860    11:37:32     85-10:32:43  0.5   512   mungr
4005    3-22:34:33   85-10:32:25  4.6   512   cobbler
4447    4-20:11:27   85-10:31:14  5.6   2048  mercury
6788    02:28:58     54-08:59:04  0.1   1024  xwiki
23916   1-03:46:02   80-09:51:47  1.4   512   fan
32539   02:17:48     1-08:50:54   6.9   1024  visualstudio

Something like the above for example, would display the name of the KVM (6th column) with the %CPU being used (4th column), along with the actual CPU time they've used thus far (2nd column) and the elapsed time they're been running (3rd column). The 5th column shows the amount of RAM allocated to the KVM.

Monitoring through the KVM Guests

If you want a more details view then you'll need to actually poke into each VM and collect the data within them locally. One way to get pretty good data is through the tool sar. Most RHEL, CentOS, and Fedora OSes have this package installed by default.

NOTE: sar is part of the sysstat package!

Without any arguments the basic sar command will return the CPU utilization for the current day, from 12AM to the current time.

For example:

$ date
Tue May  7 21:31:54 EDT 2013

$ sar
Linux 2.6.35.14-106.fc14.x86_64 (grinchy)       05/07/2013      _x86_64_        (4 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all     24.86      0.00      3.98      6.01      0.00     65.16
12:20:01 AM     all     25.12      0.00      3.98      5.98      0.00     64.93
...
...
09:20:01 PM     all     36.06      0.00      5.00      4.95      0.00     54.00
09:30:01 PM     all     35.40      0.00      4.97      4.66      0.00     54.97
Average:        all     22.29      0.08      4.37      4.80      0.00     68.46

The last line shows you the overall averages for each corresponding measure. Also notice that the data is sampled every 10 minutes. This is of course configurable.

The sar tool can do a lot more than just CPU utilization. It's a complete package and allows you to monitor file I/O, networking, memory paging, etc.

Additionally it can generate the data into CSV or XML formats (among others) and also it can generate reports too!

Package Installs

You're idea of using chkconfig --list is how I would start to determine what services are configured on a given box. Depending on how the box is setup, ours are typically set to run in runlevel 3, I'd use this command:

$ chkconfig --list |grep 3:on
abrtd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off
...
...
vboxdrv         0:off   1:off   2:on    3:on    4:on    5:on    6:off
vboxweb-service 0:off   1:off   2:on    3:on    4:on    5:on    6:off
xinetd          0:off   1:off   2:off   3:on    4:on    5:on    6:off

User's logging in

You can use the tool ac to determine if people are logging into a system and for how long.

For example:

$ ac -pd
    sam                                  0.15
Apr 28  total        0.15
    sam                                  0.32
Apr 29  total        0.32
    sam                                  0.00
Apr 30  total        0.00
    sam                                  0.01
    joeuser                              0.00
May  3  total        0.01
    sam                                  1.54
May  5  total        1.54
    sam                                  0.01
Today   total        0.01

Shows a daily breakdown of how many hours a given user was logged into a system.

Process accounting

If you're really serious about tracking what's happening on a Linux box you can enable process account. This is the service /etc/init.d/psacct. This will give you full details on when a service was last run, what commands a given user last ran, etc. It's pretty much the full nine yards.

examples

commands last run by user vivek

$ lastcomm vivek
userhelper        S   X vivek  pts/0      0.00 secs Mon Nov 13 23:58
userhelper        S     vivek  pts/0      0.00 secs Mon Nov 13 23:45
rpmq                    vivek  pts/0      0.01 secs Mon Nov 13 23:45
rpmq                    vivek  pts/0      0.00 secs Mon Nov 13 23:45
rpmq                    vivek  pts/0      0.01 secs Mon Nov 13 23:45
gcc                     vivek  pts/0      0.00 secs Mon Nov 13 23:45

last time rm was run

$ lastcomm rm
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:39
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:39
rm                S     root     pts/0      0.00 secs Tue Nov 14 00:38

For example, this last technique could be used to see the last time Apache was run (httpd).

The psacct package has many other tools as well. This is just to give you some ideas and get you started.

References

share|improve this answer
add comment

You can run

ps -C qemu-kvm -o pid,time,etime,args

on the host regularly and check the change in CPU time (the time field). That allows you an assessment over a larger period. You may save the results from time to time in order to not lose them when a VM crashes or is shut down.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.