6

I have a cloud-based server running a Java process (a POP/SMTP server called davmail, details don't really matter).

What is the simplest way to monitor the status of that Java process remotely? I need to know if and when it goes down. I could just leave an ssh session open and tail the log, but is there something more elegant?

Maybe I need to install a webserver and run some monitoring software on the same server. Or maybe there is a client program I can run on my local machine (a Mac) that can be alerted by some cron job on the server whenever the Java process stops?

  • just if and when it goes down? – strugee Apr 11 '14 at 8:07
  • I made a simple program who did something like that: it monitors if a process is executing by checking the pid file, and if it is not, relaunch again. It wakes up every 10 minutes to do that checking and run the program if needed. If you are interested I can post it now. – Raul Luna Jan 12 '18 at 9:44
3

Well you can always have a cronjob doing

echo "100 logout" | nc yourserver.fqdn 143 || \
   {
      echo "The server is down" |\
      mailx -s "Red alert! Red alert! This is not a drill!" user@domain.com;
   }

Depends on what you call simple and what other functionality you may want.

  • 100 logout is nothing but a very trivial IMAP command that, if you haven't logged in before, will make the server terminate the connection.
  • nc is a tool that opens TCP connections and attaches STDIN and STDOUT to the socket.

The combination results in establishing a connection to your IMAP server, and telling it "we're done". If your IMAP server is up and running, it recongnises the command and close the TCP connection, making nc exit normally.

If something goes wrong, for instance the TCP connection is never established, or times out because the server doesn't handle the IMAP command, your IMAP server obvously is not available. In that case you want a notification and in that case nc will always terminate abnormaly.

|| means that if nc terminates abnormaly, everyting in the { ... } is executed. The example command i provided here sends an email using mailx to user@domain.com with the subject "Red alert! Red alert! This is not a drill" and the content "The server is down".

Note however, that mailx is not part of every Linux's installation and that there are different versions of it, behaving differently.

You can, of course, deploy some monitoring software, for instance Shinken, which have a fancy web frontend, keep track of what they monitored and can send emails.

  • Omg, just noticed IMAP is not what you have. I promise to update the answer for SMTP and pop. Its simply other "we're done" messages – Bananguin Apr 11 '14 at 16:31
2

That all depends on what you mean by "goes down", the details of what you are running and doing do tend to matter when monitoring as well....

The most thorough form of monitoring for your "service" is to have an external automated system do what your clients do and report to you when anything unexpected occurs.

From the brief description of your email service, my first end to end test would be:

  • Send an email #id via SMTP to a local monitor account
  • Wait the maximum amount of seconds you want that to take.
  • Check for email #id @ monitor on the POP server.

That one check from an external monitoring host is going to pick up about 99% of the problems that could occur in a simple email system.

These service or transaction monitors tend be custom scripts written in something like Ruby, Python or Perl which have modules to easily implement things like SMTP or POP programatically. The scripts then generally plug into a monitoring solution but even triggering a simple email or SMS via a gateway from a cron job would suffice if you want something simple. If you pay for a monitoring solution you will generally get their attempt at a GUI designer for the same type of monitor.

Of course in the real world this quickly becomes more complicated. You might offer secure ports for both POP and SMTP requiring another check. Maybe IMAP is added in with a bit of Carddav and Caldav, you might have services on multiple hosts.

What an overall service check like the above won't readily tell you, is where the problem lies, just that there is a problem somewhere.

Lower level monitoring

By monitoring individual components of the you service you basically make it easier to identify (or predict) where an issue lies from monitoring before doing any leg work. This type of component monitoring is what systems like Nagios, Zabbix or big ones like Tivoli Monitoring are good at.

This can be an ever expanding tree of things, depending on how detailed you make it and again, how complex the system is that supports your "service"

"Your mail Service" depends on

Services:     POP:110 SMTP:25
Application:  devmail
OS:           linux Z
Host:         server Y 
  Components:   diskA diskB cpu1 cpu2  memory          
Ntwork:      ethernetA, Switch B, Router C, Firewall X

Each component has metrics or a state you can report on.

Externally

Service: 
  POP service   - Are we accepting connections on 110,995
  SMTP service  - Are we accepting connections on 25,587

Locally

Application: 
  devmail process(es) (is it running, memory, cpu, handles, io)
  JMX parameters of the java process (memory, threads, performance, garbage collection)
OS: 
  Disk, Memory, Cpu, IO

etc...

What if the monitoring host goes down? Or it's just the network between the service and monitor.

It's generally good to run the service checks from two (or more) external hosts which are as close to where the clients are coming from (without impacting your monitoring). Then also run the checks locally on the host or at least from local network. This way you get a better idea of most network based problems.

  • If one external client fails, probably an external network.
  • If local client is working, but all external clients are failing, probably local network.
  • If all clients are failing, probably local issue.

I see a lot of people tend to build their monitoring solutions the wrong way around. They come up with a lot of the lower level system metrics and 1000's of monitors and level's they think are appropriate to alarm at, when none if that really matters. I mean they're nice for analysis and capacity management, you can make some swanky graphs out of all those values and they can be extremely useful but don't really mean much when you missed metric x at level y that means no one can receive emails.

1

There are any number of monitoring applications that can be configured to send an alarm in various manner - by SNMP traps, by email, by SMS if you've got the hardware or software/subscription. Many of them can also restart the process on itself as necessary.

Google for monit, nagios, or just "monitoring software". Also https://softwarerecs.stackexchange.com/ may be a good place to ask for recommendations.

0

Using nagios or monit locally is hard to avoid but you can get a great second line of defense by using something like https://cronitor.io to monitor the nagios process itself. Setup nagios or monit to do a health check against a Cronitor address and you'll be alerted if the health check doesn't run.

0

I'd suggest trying out SeaLion. It is a cloud based Linux server monitoring tool. Installation takes just seconds and UI is clean and simple. Plus the alerting feature is great. There is a 'daily digest' feature which sends you a timed email everyday with the summary of the servers' performance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.