I'm just jumping into unix from a different world, and wanted to know if

while true
do
  /someperlscript.pl
done

The perl script itself internally has a folder/file watcher that executes when files are changed in the target location.

Is this (while true) a good idea? If not, what is a preferred robust approach?

TIA

EDIT : Since this seems to have generated a fair bit of interest, here is the complete scenario. The perl script itself watches a directory using a file watcher. Upon receiving new files (they arrive via rsync), it picks up the new one and processes it. Now the incoming files may be corrupt (don't ask.. coming from a raspberry pi), and sometimes the process may not be able to deal with it. I don't know exactly why, because we aren't aware of all the scenarios yet.

BUT - if the process does fail for some reason, we want it to be up and running and deal with the next file, because the next file is completely unrelated to the previous one that might have caused the error.

Usually I would have used some sort of catch all and wrapped the entire code around it so that it NEVER crashes. But was not sure for perl.

From what I've understood, using something like supervisord is a good approach for this.

  • 9
    Is this Linux or UNIX? If it's Linux you might want to check the inotify API so that you can avoid a busy loop waiting for files to change in your target directory. – roaima Mar 30 '15 at 10:52
  • Its linux, ubuntu – Abhinav Gujjar Apr 1 '15 at 13:04
  • If you know the file is good before it leaves the pi, then you could just run a checksum like md5 or sha1 and send that along with your file. Then the receiver will know if it got a bad file before it tries to process it. If you don't know that, then you can still build some sort of partial or block checksums or something similar into your data file which insures data integrity, so you can check things as you go before you commit to a process which can fail. An ounce of prevention ... – Joe Apr 3 '15 at 22:47
up vote 20 down vote accepted

That depends on how fast the perl script returns. If it returns quickly, you might want to insert a small pause between executions to avoid CPU load, eg:

while true
do
  /someperlscript.pl
  sleep 1
done

This will also prevent a CPU hog if the script is not found or crashes immediately.

The loop might also better be implemented in the perl script itself to avoid these issues.

Edit:

As you wrote the loop only purpose is to restart the perl script should it crashes, a better approach would be to implement it as a monitored service but the precise way to do it is OS dependent. Eg: Solaris smf, Linux systemd or a cron based restarter.

  • 8
    You could do while sleep 1; do ... to save the true call. – Raphael Ahrens Mar 30 '15 at 15:22
  • 3
    @RaphaelAhrens Indeed although that would slightly change the initial behavior. The alternate until ! sleep 1; do ...; done will still save that builtin call while starting the script immediately. – jlliagre Mar 30 '15 at 18:29
  • 4
    Totally, totally agree that a hot loop should be avoided with a sleep call inside the loop if you're going to do this. Also agree that an event-driven script (inotify, et al) would be a better solution. But using a while loop is not intrinsically evil, necessarily, unless it's infinite and hot. I think the more important issue is probably dealing with why the perl script fails and needs to be restarted, though. – Craig Mar 31 '15 at 3:08
  • 2
    Better: sleep 1& /someperlscript.pl; wait. – user23013 Mar 31 '15 at 10:13
  • 2
    @EliahKagan Well spotted. TBH, I never use the untilinstruction, I have confused it with an hypothetical do/until loop which doesn't exists in the shell. – jlliagre Dec 10 '17 at 21:54

The other answers, about using inotify, are correct, but not an answer to this question.

A process supervisor, such as supervisord, upstart, or runit, is designed for exactly the problem of watching and restarting a service if it crashes.

Your distro probably comes with a process supervisor built in.

while true is fine as a general-purpose "loop forever" construction. As other answers say, the body of the loop shouldn't be empty, or become empty by virtue of the command inside the loop not working.

If you're using Linux, you may want to use a command like inotifywait, which makes the while loop much simpler:

while inotifywait -qqe modify "$DIRECTORY"
do
    process_the_directory "$DIRECTORY"
done

Here, the inotifywait command will sit and wait for a filesystem event to occur (in this example, when files in the directory are written to). At that point, it exits successfully and the body of the loop executes. It then goes back to waiting again. Because the inotifywait command waits for something to happen in the directory, it's much more efficient than continually polling the directory.

  • ` (apt-get or yum) install inotify-tools` +1 cool! – JJoao Mar 31 '15 at 12:32

Move the while 1 into the perl script (Following @roaima suggestion)

#!/usr/bin/perl

 use Linux::Inotify2;

 my $inotify = new Linux::Inotify2 or die "unable to inotify: $!";

 $inotify->watch ("Dir", IN_MODIFY, ## or in_{acess,create,open, etc...}
   sub { my $e = shift;
     my $name = $e->fullname;
     ## whatever 
     print "$name was modified\n" if $e->IN_MODIFY;
  });

 1 while $inotify->poll;
  • 1
    This doesn't address the restart requirement should the script crashes or is killed for some reason. – jlliagre Mar 30 '15 at 11:49
  • @jlliagre, thank for the comment. In the answer, I don't see a clear specification for the intended behavior after a crash or kill. This is clearly a interesting and relevant question. For the moment I will keep it simple (if the script died or was killed, stay dead :) – JJoao Mar 30 '15 at 12:19
  • @jlliagre That's what exceptions are for. But Perl might be not the most appropriate choice in such a case since it doesn't support the exception mechanism. Just a guess. Would be best to port the script to a language that supports exceptions. It all depends how big the porting work is, of course. – user86969 Mar 30 '15 at 12:20
  • @Nasha, In Perl, with signal handlers, error variables, Tyr::Tiny, etc, we can do it! ... but -- I hate exception handling and error recovery: they are never complete... – JJoao Mar 30 '15 at 12:35
  • 1
    @JJoao I agree with you on the facts that error recovery is rarely complete. It is of course up to the developer to cover all possible cases. So is it with PLCs in the industry world therefore it must be quite possible after all ;-) . – user86969 Mar 30 '15 at 12:48

When your perl script is intended to keep on running all the time, why use the while construction ? When the perl fails in view of some serious problem, the new perl script started by the while might crash just as hard. Again-and-again-and-so-on.
if you really want your perl started over, consider crontab and a script that first checks for running instances. This way your script will even be started after a reboot.

In general there is no problem using while true since it is a tiny test which is only executed after the perl script is terminated. Keep in mind that depending on which Linux/Unix variant you are using, the script might get terminated at logging off. In such case consider using the loop in a script and call it with nohup and put it in the background i.e. nohup myscript &

If the perl script terminates too often and causes CPU load than this load is accountable to the perl script and not the while true.

See also man nohup for details.

  • The only problem is the potential for a hot loop that drives up CPU utilization. So I totally agree with putting a sleep call inside the loop to tame it down. – Craig Mar 31 '15 at 3:00

If you want to manage a process, you might want to look into a process manager to do so.

With many recent systems, you can use systemd to monitor your processes (which is one of the benefits of systemd over classical init-scripts). If your distro of choice doesn't use systemd, you could use daemontools or monit.

  • mon is a simple alternative, if the lumbering hugeness of monit and systemd bother you as much as they do me. – Anko Mar 31 '15 at 12:44

The while loop is really not a good idea. There's no escape there - it just runs forever - statically. Any number of things could change in the environment and it will not be affected - and this could be bad.

For example, if the shell executable responsible for that while loop is upgraded the kernel will not be able to release the disk space for the old version until that script quits because it needs to maintain the descriptor as long as it runs. And that is true for any files the shell might have open for whatever reason to run the loop - they will all be held open by this while loop for as long as it runs - forever.

And if, heaven forbid, there is a memory leak anywhere in the shell running that loop - even the most minute - it will simply continue to leak - statically. It will build unchecked and the only recourse is to forcefully kill it, then start it over just to do the same later.

This really isn't the way you should setup a backgrounded process - at least, not in my opinion. Instead, as I think, there should be a reset point - a refresh of the script and its state. The easiest way to do this is with exec. You can replace the current process with a new one - while maintaining the same PID - but still running a new process.

For example, if your perl script should return true after it has successfully handled a file modification in some monitored directory:

#!/bin/sh
trap 'rm -rf -- "${ldir%%*.}"' 0 INT
_exec() case    $#      in
        (0)     exec env -  "PID=$$" "ldir=${TMPDIR:-/tmp}/." \
                            "$0" "$@";;
        (*)     export "$@" "boff=0" "lmt=30"
                exec        "$0" "$@";;
        esac
[ "$PID" = "$$" ] || _exec
[ -w "$ldir" ]  &&
case    $ldir   in
(*.)    until   mkdir -- "$ldir"
        do      :& ldir=$ldir$$$!
        done    2>/dev/null
;;
(*)     until   /someperlscript.pl ||
                [ "$((boff+=1))"  -ge "$lmt" ]
        do      [ -d "$ldir" ]     &&
                sleep "$boff"      || ! break
        done
;;esac  &&      _exec ldir PID

...or something like that. Something that allows the basic machinery behind the loop to refresh every once in a while.

I upvoted a few of these answers, but I instinctively have the warmest feelings for @WalterA's answer. Well, I did until I created my own answer...

Personally, I'd tend to update the Perl script so that it writes a descriptive log entry about the failure and sends an alert to an administrator.

If the Perl script is meant to continue running, waiting for change events from the file system, why is it failing?

If it isn't failing, why are you concerned about wrapping it in a script to restart it infinitely?

If there's a configuration issue or some broken dependency that causes the Perl script to abort, simply restarting it over and over isn't likely to make it suddenly start working.

You know the definition of insanity, right? (doing the same thing over and over again, expecting a different result). Just saying. ;-)

  • haha- I know, i know. but you know - the real world sucks. Anyways - the reason is that it processes files, and some files may not even get transmitted correctly. We don't yet know the variety of reasons it may fail. I see your point about logging the error, but I will still need it backed up to process the next file via something like supervisord – Abhinav Gujjar Apr 1 '15 at 13:13
  • If you can catch the exceptions, you can log them then keep processing, and even go back and occasionally retry failed files? Maybe the the failed file was locked by another user or process when you tried to process it, etc. – Craig Apr 1 '15 at 17:27

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.