4

During monitoring of SunOS5.10 machine I noticed strange behavior:

  1. Disk busy is close to 100%
  2. Disk waiting time is 0,0%

Primary purpose of this machine is running application server on jvm. This situation happens when server is under heavy load and clients connections are randomly rejected. If problem is IO related probably IO wait time should be more than 0,0%. I can't see any other bottlenecks on the machine: cpu, ram are plenty and network interfaces are utilized under 5%. At the time of disk saturation system writes around of 0,5 MB of data per second and reads almost none.

Data is dumped with sarmon, and iostat -x is used for live monitoring. Sample iostat output:

extended  device  statistics
device  r/s    w/s    kr/s  kw/s  wait  actv  svc_t  %w  %b
sd0,a   0.0    0.4    0.0    6.2   0.0   0.0   11.5   0   0
sd0,b   0.0    0.0    0.0    0.0   0.0   0.0    0.0   0   0
sd0,c   0.0    0.0    0.0    0.0   0.0   0.0    0.0   0   0
sd0,f   0.0    0.2    0.0    1.6   0.0   0.0   13.9   0   0
sd0,g   0.0    0.0    0.0    0.0   0.0   0.0    0.0   0   0
sd0,h   0.0  111.6    0.0  519.9   0.0   1.0    8.9   0  94

Statistics are taken at the OS level, so measurements should be correct. Is it possible on the application level to reliably detect disk saturation and reduce load accordingly?

What are possible scenarios when such situation occurs?

I have also used dd to test write speed of the disk with the following results:

time sh -c "dd if=/dev/zero of=/tmp/output bs=512 count=2048k"
2097152+0 records in
2097152+0 records out
real 0m3.250s
user 0m0.627s
sys  0m2.622s
  • What command(s) do you use? Please post a sample output. – jlliagre Nov 25 '14 at 11:50
  • Data is dumped with sarmon, and iostat -x is used for live monitoring. Sample iostat output: extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b sd0,a 0.0 0.4 0.0 6.2 0.0 0.0 11.5 0 0 sd0,b 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd0,c 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd0,f 0.0 0.2 0.0 1.6 0.0 0.0 13.9 0 0 sd0,g 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sd0,h 0.0 111.6 0.0 519.9 0.0 1.0 8.9 0 94 – m.kowalski Nov 25 '14 at 12:24
  • I have also used dd to test write speed of the disk with the following results:time sh -c "dd if=/dev/zero of=/tmp/output bs=512 count=2048k" 2097152+0 records in 2097152+0 records out real 0m3.250s user 0m0.627s sys 0m2.622s – m.kowalski Nov 25 '14 at 12:34
3

There looks to be a single thread writing a little bit more than 4KB about 111 times per second. This is sufficient to keep your disk 100% busy (111 iops * 9 ms service time = 1 second of service per second = 100%). As there are no other processes writing on that disk (that partition actually), the wait queue is empty, all requests are processed immediately.

There is then nothing special or strange in your workload, the disk is the bottleneck.

To get better performance, you might pick a faster disk or an SSD, or use use an array of disk that distribute writes, or tune the application for it to use larger blocks, or increase the cache size.

Your dd test is pointless, you are not measuring the disk performance but essentially the cache one, especially on Solaris where /tmp is likely to be backed by virtual memory, tmpfs being the default for that file system.

1

http://en.wikipedia.org/wiki/Asynchronous_I/O, a program is firing up requests for I/O, but does not wait on them. However it can still accept and process the I/O responses once they come.

  • How do you think this link answers the OP question ? – jlliagre Nov 26 '14 at 9:44
  • the question was "What are possible scenarios when such situation occurs?" – Thorsten Staerk Nov 26 '14 at 13:25
  • Asynchronous I/Os cannot be a valid explanation of zero iowait. That might be at best the opposite. – jlliagre Nov 26 '14 at 13:34
  • oh, you mean because there will still be shadow processes waiting for the blocks? That could be, I did not think so far, sorry. – Thorsten Staerk Nov 26 '14 at 13:39
  • I don't get what you mean with shadow processes waiting for blocks. There are only writes in the OP case anyway. The iowait metric is reporting the percentage of time during while the wait queue is not empty. This queue is not empty when there at least an I/O is ready to be processed but the disk cannot handle it immediately being busy doing something else. Enabling asynchronous I/Os will increase the risk for that queue to be filled, not the opposite. – jlliagre Nov 26 '14 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.