Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to create a crawler that scans one single web page and sends me an email in case something is wrong on the page (the page holds some webservers' status - Up/Down).

What would be the best language to write such a service in?

I've considered PHP or Python, but as they are interpreted languages (+PHP has a max time limit that a script runs), I don't think they're the best fit for this.

Also, C/C++ is not an option. I was thinking about Java, since it has Threading and I may need it, but I'm not sure.

share|improve this question

closed as not constructive by rahmu, uther, Gilles, jordanm, jasonwryan Feb 5 '13 at 19:18

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

If bash is not enough (curl http://example.com/ | grep -q 'Up' || mail -s 'Warning' <<< 'Site not up') the specification you posted is probably incomplete. – manatwork Feb 5 '13 at 13:32
It's not really enough, because I want some more advanced stuff like sending an email only if the server is down more than X seconds, and not send the email twice if a server is still down. – Eduard Luca Feb 5 '13 at 13:43
So you want to write your own monitoring system? None of the tons of existing monitoring services such as nagios or zabbix do the trick? – jordanm Feb 5 '13 at 13:53
In that case my sketched solution is not enough, but I still find bash just enough for the task. – manatwork Feb 5 '13 at 13:53
Extend Xymon (xymon.sourceforge.net) if needed with BASH scripts. (Probably no need.) It's easy, ready made, free cost, open source. – Christopher Feb 5 '13 at 14:24
up vote 3 down vote accepted

Just use the language you are the most comfortable with. This kind of application is not CPU-/memory-bound.

You were mentioning Python. It is interpreted but I really don't see why this would be a problem. Youtube/Tornado/Django/etc. run Python. All the things that you will ever want for your specific purpose are already available as modules (daemonize, http, etc.).

PHP does not seem to be a good fit because of the limit that you mention, but also because you need apache (not completely true, but whatever). Still, it is possible.

Java would also perfectly fit. You can also daemonize your process and use existing libraries.

C/C++ seems a bit to much, but with the available libraries, you should be able to generate some working code very quickly.

As manatwork stressed, bash also just fits...

Well, you guessed it, my point here is that for this kind of application, there is no "best language". There is a family of language which already implement a http_client library of some sort and the others. Your CPU and memory are not limiting. You can just enjoy the language you want. :) But generally speaking, if this is your server, checking your web service is up should be checked internally (cron job, nagios, etc.). From your home you may just want to ping the server to see if the hardware is up and responding.

share|improve this answer
Thanks! Will either go with an already built solution like the comments said, or Python. – Eduard Luca Feb 6 '13 at 9:40

What I would do is write it in a scripting language such as Perl or Python and then scale up to Java if need to in search of features such as better concurrency management. You always want to use the simplest tool for the job and scale up if you need to. I am a huge fan of both languages and even though I know Java better, I would always try to solve any problem in Perl before resorting to Java. Perl is a great candidate for offline workload processing, especially single-threaded.

Yet another question to be asked is the availability of APIs for the particular application, although nowadays there are APIs and web services for just about anything. Still, I would bet there are more APIs for Java than Perl. I was, however, recently disappointed not to be able to find a Java API for IBM Clear Case. So for some version management stats and monitoring, I used shell invoking their CLI and passing data to Perl for further processing.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.