I have one process running on a RHEL 6.3 machine. Somehow this is taking a longer processing time than the same process running on another machine. How is it possible to know why this process is running slow or if any other parallel running process is slowing down this process?
You could try debugging what's going on by using either
htop show the amount of resources a particular process is consuming. I would run your process like this:
/usr/bin/time myproc to see how much time it's really taking to start. This command should show the same user time but different elapse times across the 2 machines.
Once you've established that the process is taking the same "effective" amount of time on the various systems you can then use either
htop to determine what "other" processes are slowing the system down so that this process is taking longer on machineA vs. machineB.
Say I have the following script,
#!/bin/bash a=0 while [ 1 ]; do let a=a+1 (( $a==100000 )) && exit done
I then run it on unloaded system:
$ /usr/bin/time ./doit.bash 1.25user 0.10system 0:01.38elapsed 98%CPU (0avgtext+0avgdata 1272maxresident)k 0inputs+0outputs (0major+366minor)pagefaults 0swaps
I then run this command to simulate some load on my box:
$ dd if=/dev/zero of=/dev/null
Repeating a run of
$ /usr/bin/time ./doit.bash 2.32user 0.20system 0:02.98elapsed 84%CPU (0avgtext+0avgdata 1268maxresident)k 0inputs+0outputs (0major+365minor)pagefaults 0swaps
You can see that
doit.bash took longer to run in both terms of the total user time and elapse time.
- run1 - (user): 1.25 seconds - (elapse): 1.38 seconds
- run2 - (user): 2.32 seconds - (elapse): 2.98 seconds
NOTE: make sure to Ctrl + C the running
dd command when you're done.
Do you have any idea if the process is CPU or IO limited? If it needs only CPU,
top is your friend. If it's IO limited, check out
vmstat -Sm 5,
iotop -od5 and
iostat -m 5. Try to figure out if the problem is caused by slow poor device performance or if there's something else running on the worse performing system.
If you don't know if it's CPU or IO limited, start with
top and look at the system IO wait value (
wa) at the top. If one or more CPUs are waiting for IO, the IO wait value will raise. If you have lots of idle, and low IO wait value, then your process is waiting for some kind of syncronization events (probably some kind of locking either locally or over network).