2

I'm not an expert in Solaris, been used to Windows OS. Anyways, I'm having a hard time trying to find out an issue that has been going on, but can't find an answer.

We have a physical server with Solaris 11, that has 3 LDOMs w/ Solaris 10. Each LDOM has one zone (besides the global one, global zone doesn't have any configuration at all). That Zone is a Solaris 8 (this is due to the applications that are run under it that not supports a Solaris version higher than 8)

Now we have problems with the Zone, the DB is in one disk, while software and other stuff are in different disks. Users are complaining that the server feels slows.

When I check the status with top and iostat, things look like this

load averages:  1.82,  1.74,  2.71                                                                             09:45:06
1047 processes:1040 sleeping, 2 zombie, 2 stopped, 3 on cpu
CPU states: 85.0% idle, 11.5% user,  3.5% kernel,  0.0% iowait,  0.0% swap
Memory: 56G real, 12G free, 25G swap in use, 8798M swap free

[image]

The highest value for load has been

5.xx 6.xx 
CPU States: 40% idle,
Memory: 4G free

While the iostat results show

root # iostat -xtc
                  extended device statistics                      tty         cpu
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b  tin tout  us sy wt id
vdc0         0.3    1.0    4.4   14.2  0.0  0.0   33.1   0   0    2  112  141 172  0 162
vdc1        40.9    3.6  667.9   78.7  0.0  0.2    3.4   0   8
vdc2         2.0    1.0  127.1    5.1  0.0  0.0    2.7   0   0
vdc3         0.0    3.8    0.0   90.9  0.0  0.0    3.8   0   1
vdc4        62.6   31.5 17615.7 1232.5  0.0  7.4   78.9   1  82
vdc5        12.5    7.9  281.2  421.3  0.0  0.1    7.2   0   4
vdc6         0.0    0.0    0.0    0.0  0.0  0.0    2.8   0   0
vdc7         0.0    7.3    0.0  451.0  0.0  0.0    2.1   0   1
vdc8        40.6    3.6  667.9   78.8  0.0  0.1    3.3   0   8

[image]

Disk 4 (vdc4), where the DB is located, has a high percentage of %b all the time and always at least has 1 process in waiting (%w), not sure if it looks bad, but considering that more than 150 users access it, I consider it to be OK. Correct me if I'm wrong

Now, whenever the user X is even listing or pressing enter in the CMD, the server takes long to show the new entry, it doesn't have issues with login, they actually login quickly through ssh. The weird thing is that root user is working just fine when they are complaining. It doesn't matter if server is low or high on resources, the same issue always happens.

Checking what is running the user this are its only process.

# ps -fu user
     UID   PID  PPID   C    STIME TTY         TIME CMD
user       6027  6024   0 08:13:14 pts/15      0:00 -ksh
user       186   181   0 09:40:48 pts/4       0:00 -ksh
user       555 15455   0 09:42:52 ?           0:00 in.ftpd
user       14114 14104   0 08:42:06 pts/7       0:00 -ksh
user       24325 14114   0 09:15:28 pts/7       0:00 tail -f XXXXXXXX
user       26 15119   0   May 30 ?           0:35 ./oplinkse_SGCR6
user       8412 15119   0 01:59:24 ?           0:01 XXXXXXXXXXXXXXXXXXXX
user       27    26   0   May 30 ?           7:00 ./oplinkse_SGCR6
user       1504  6027   0 09:46:24 pts/15      0:00 tail -f XXXXXXXX
user       5818  5815   0 08:12:39 pts/14      0:00 -ksh

[image]

They are only viewing some files and are connected to the DB thru 2 openlink sessions. Even when they are not running anything and just want to ls -l a directory that has 3 files it takes long (even 1min sometimes)

What could be checked to find out the problem?

I've lookup thru the internet, but anything I find out is regarding the slow login prompt thru SSH for users and that is not something that happens here, cause they get the login prompt right away, but after the login when they one to execute a command it stays there for a lot of seconds.

9
  • 3
    We prefer to have text posted as text rather than as images. Why did you reject JasonD’s suggested edit, after he went to all the effort of transcribing your screenshots for you? Commented Aug 2, 2017 at 15:18
  • 2
    What update of Solaris 10 and which patch level ? extended mapin space activated on control and guest domain? BTW: How many disks are hiding between vdc4 ? Is vdc a real disk or a emulated volume by ZFS? Is the vdc the only user of the spindles (for example as other LUNS are represented by the same set of disks). You are reading with 284 Kbyte block size ? What is on this device vdc: Database with raw disk, zfs, ufs ? The possible reasons are legion ....
    – c0t0d0s0
    Commented Aug 2, 2017 at 17:33
  • Can you provide an example of what command(s) are slow for users? Also, the first line of iostat output is usually cumulative since the iostat counters were last reset, usually at the last reboot. Could you post the output of iostat -sndzx 2 - make sure you get at least two sets of output. And tell us what the actual physical disk(s) are for vdc4, as whatever it is has been averaging close to 100 IO operations/sec almost certainly since the last reboot. Commented Aug 3, 2017 at 9:08
  • Also, run echo ::memstat | mdb -k as root and post the output. And don't post it as an image - post the actual text. Commented Aug 3, 2017 at 9:14
  • @AndrewHenle any command, even if they do "ls" to list a directory containing 4 files, or a "pwd" the output takes anytime from 20sec to 1min to show. If I do it with root it doesn't happen
    – cees09
    Commented Aug 3, 2017 at 15:05

2 Answers 2

1

First, can you expand on: "Users are complaining that the server feels slows." Some of your wording hints at a network lag while others hint at app slowness.

Since you're using LDOMs (now Oracle VM for SPARC), you must be using a SPARC server. The hardware, v11 and LDOM release, would also be useful. You'll also want to give the configuration for each LDOM. Perhaps a configuration issue?

I'm also wondering if you might get better performance by only having (1) Solaris 10 LDOM on your v11 system (which I assume can't run v10), with (3) Solaris 8 branded zones. Vs your current config of (3) ldoms each running (1) Solaris 8 branded zone.

Useful link on running branded Solaris 8 zone?

3
  • Mostly what feels slows is the fact that anytime the try to do a command ls or pwd (for example) the server takes time to show the ouput. I doubt is the connection since we are in the same vlan as the users, when I use root those issues don't arise (though we have asked telecom support to check the network performance and not issues has been detected) as for the running 3 branded zones S8 on 1 LDOM, hard to tell, since this is a production server I doubt we would be allow to make changes. Each zone host multiple services that needs to be up 24x7.
    – cees09
    Commented Aug 3, 2017 at 15:15
  • # ldm -V Logical Domains Manager (v 3.2.0.1.3) Hypervisor control protocol v 1.12 Using Hypervisor MD v 1.4 System PROM: Hostconfig v. 1.4.3 @(#)Hostconfig 1.4.3 2015/04/10 09:55 Hypervisor v. 1.14.2.a @(#)Hypervisor 1.14.2.a 2015/05/06 15:22 OpenBoot v. 4.37.2 @(#)OpenBoot 4.37.2 2015/04/10 08:59
    – cees09
    Commented Aug 3, 2017 at 15:16
  • root@proclu2:~# pkg info ldomsmanager Name: system/ldoms/ldomsmanager Summary: Logical Domains Manager Description: Oracle VM Server for SPARC - Virtualization for SPARC M-Series and T-Series Category: System/Virtualization State: Installed Publisher: solaris Version: 3.2.0.1.3 Build Release: 5.11 Branch: 0.175.2.11.0.3.0 Packaging Date: May 26, 2015 02:43:48 PM Size: 4.52 MB /[email protected],5.11-0.175.2.11.0.3.0:20150526T144348 which config info could I provide you?
    – cees09
    Commented Aug 3, 2017 at 15:16
0

This isn't really an answer, but a comment won't work for this, and it is a troubleshooting tool that can help get an answer.

This DTrace script will give a good indication of where your system's kernel is spending its time:

#!/usr/sbin/dtrace -s

#pragma D option quiet

profile:::profile-1001hz
/ arg0 /
{
    @hot[ arg0 ] = count();
}

dtrace:::END
{
    printa( "%@u %a\n", @hot );
}

It captures many samples of all kernel threads' current function, so if your system is spending a lot of its time performing a small set of tasks, this little script will reveal that very quickly.

To see the actual kernel stack, you can use

#!/usr/sbin/dtrace -s

#pragma D option quiet

profile:::profile-1001hz
/ arg0 /
{
    @hot[ stack() ] = count();
}

dtrace:::END
{
    printa( "%@u %a\n", @hot );
}

Save it to a file such as hot.d, make the file executable with something like chmod 755 hot.d, then, as root, run it: ./hot.d. It won't emit any output. Let it run for a good while, such as 30 seconds or so. Then hit CTRL-C to stop it. It will then emit all the sampled kernel current function or stack traces that it encountered when it was running, in ascending order of how many times that specific stack trace was observed.

The last few functions or stack traces in the output output will likely reveal what your system is spending most of its time doing.

For example, if your kernel is spending most of its time doing something such as coalescing fragmented memory pages into the large pages required by an Oracle database, you'll see it immediately.

Run it in your Solaris 11 hypervisor, then in each of the global zones.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .