1

I have an OpenNMS Linux server which is having poor disk performance. The machine is a EC2 inside a VPC. atop shows:

enter image description here

iotop shows:

enter image description here

How do I reduce the IO numbers and bring down the disk utilization??

ionice response:

ubuntu@ip-10-12-251-11:~$ sudo ionice -c3 -p $(pidof opennms)
ionice: option requires an argument -- 'p'

ionice - sets or gets process io scheduling class and priority.

Usage:
  ionice [OPTION] -p PID [PID...]
  ionice [OPTION] COMMAND

Options:
  -c, --class <class>   scheduling class name or number
                           0: none, 1: realtime, 2: best-effort, 3: idle
  -n, --classdata <num> scheduling class data
                           0-7 for realtime and best-effort classes
  -p, --pid=PID         view or modify already running process
  -t, --ignore          ignore failures
  -V, --version         output version information and exit
  -h, --help            display this help and exit
5
  • EC2 are not famous for I/O capabilities; nevertheless how many objects have you in your NMS? We have 3,000 objects in our Nagios, and not a single problem. What you have to have in mind is that you cannot do all the probes in a 5-minute window with that many objects. Commented Apr 13, 2016 at 7:08
  • i am guessing a DISK size increase may tweak this.. not sure though coz i have 2 other EC2 instances which run openNMS in diff region with 100gbs , while this instance is having 8gbs only Commented Apr 13, 2016 at 7:10
  • 1
    As a rule of thumb, it is better to have around 70% of the disk free. Commented Apr 13, 2016 at 7:12
  • yep. absolutely , I am gonna fix this and close this question when done Commented Apr 13, 2016 at 7:16
  • The ionice fail is telling you that the -p option needs parameters (PIDs in this case) but isn't seeing any. That means that pidof opennms returned nothing, and that means opennms wasn't running at the time.
    – agc
    Commented Apr 13, 2016 at 10:29

2 Answers 2

1

Try rrenice from Debian's pslist package. Example, this sets the priority of opennms and its descendants to the lowest possible setting:

sudo rrenice 19 opennms

Or that's not available, use plain renice:

sudo renice -n 19 -p $(pidof opennms)

For disk hog programs, use ionice:

ionice -c3 -p $(pidof opennms)

BTW: that opennms process shouldn't be that resource hungry. Something's either buggy or hung there.

9
  • ionice -c3 -p pidof opennms` ` this doesn't work that ways.. shows me how to use Commented Apr 13, 2016 at 7:08
  • @NishantSingh, please check that both ionice and pidof exist. Run: which ionice pidof
    – agc
    Commented Apr 13, 2016 at 7:20
  • returns /usr/bin/ionice /bin/pidof Commented Apr 13, 2016 at 7:26
  • how to know the process id of openNMS Commented Apr 13, 2016 at 7:29
  • @NishantSingh, that's what *pidof does, see man pidof. Anyway you have both ionice and pidof, so let's try one more time, without backticks: ionice -c3 -p $(pidof opennms)
    – agc
    Commented Apr 13, 2016 at 7:32
1

Those figures show a write-bound process, so the cache tuning suggested by OpenNMS won't help.

However it could help if you sacrificed durability. If your OpenNMS dies, it doesn't really matter if the database shows it dying a few seconds earlier. (This is unlike e.g. an email server, where clients are relying on at-least-once semantics). You could lose configuration changes made immediately before a crash though; maybe just make sure you notice crashes, and check the system is still online after making config changes. Try synchronous_commit = off. By default, the window of durability loss is just 600ms.

If synchronous_commit = off has the desired effect, there's a possible alternative that doesn't sacrifice durability. See commit_delay. It seems to me that OpenNMS could work very well with that too. You'd want to set a commit_delay on the order of the reciprocal of your observed IOPS. Then I think you could increase <poller-configuration threads= if CPU (and RAM) utilization is still low.

3
  • where do i found the mentioned settings? Commented Apr 13, 2016 at 8:38
  • Somewhere under /etc/postgresql. I don't know if there'll already be a file that sets them or not. I think you can put them in any .conf file in the right directory.
    – sourcejedi
    Commented Apr 13, 2016 at 9:17
  • And OpenNMS poller configuration is apparently called poller-configuration.xml. opennms.org/wiki/Polling_Configuration_How-To
    – sourcejedi
    Commented Apr 13, 2016 at 9:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .