I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's

for file in ./folder/*;
    ./bin/myProgram $file > ./done/$file

(Please Ignore syntax errors, it's just pseudo code).

I now wanted to run this script twice at the same time. Obviously, the execution is unnecessary if ./done/$file exists. So I changed the script to

for file in ./folder/*;
    [ -f ./done/$file ] || ./bin/myProgram $file >./done/$file

So basically the question is: Is it possible that both scripts (or in general more than one script) actually are at the same point and check for the existance of the done file which fails and the command runs twice?

it would be just perfect, but I highly doubt it. This would be too easy :D If it can happen that they process the same file, is it possible to somehow "synchronize" the scripts?

  • If you have a version of xargs with the -P option available, see this question. – jw013 May 10 '12 at 8:58
  • 2
    GNU Make supports parallel execution, too; the done/$file markers seem a little like make targets to me. – sr_ May 10 '12 at 9:11
  • 2
    The (pseudo-)code you posted doesn't actually run two instances of your program in parallel. If you have either xargs or GNU make or some version of parallel, then there is no need to reinvent this particular wheel. – jw013 May 10 '12 at 9:20
  • It will run two instances if the above script is executed twice – stefan May 10 '12 at 9:22

This is possible and does occur in reality. Use a lock file to avoid this situation. An example, from said page:

if mkdir /var/lock/mylock; then
    echo "Locking succeeded" >&2
    echo "Lock failed - exit" >&2
    exit 1

# ... program code ...

rmdir /var/lock/mylock
| improve this answer | |
  • 1
    "As an aside, you almost definitely want to quote $file." It wasn't necessary for my simple job, but obviously your right, that it would be better to do so – stefan May 10 '12 at 9:02
  • @stefan - I removed it once I saw "ignore syntax errors"... :-) – Chris Down May 10 '12 at 9:04
  • :D it's perfectly fine if you remind me of this stuff. I tend to forget it since I'm not yet used to it – stefan May 10 '12 at 9:08
  • I absolutely LOVE the simplicity of the mkdir-locking. Thanks for the link! (even though I would have preferred it reading about it on this site, maybe you want to extend your answer a bit?) – stefan May 10 '12 at 9:11

The two instances of your script can certainly interact in this way, causing the command to run twice. This is called a race condition.

One way to avoid this race condition would be if each instance grabbed its input file by moving it to another directory. Moving a file (inside the same filesystem) is atomic. Moving the input files may not be desirable, and this is already getting a bit complicated.

mkdir staging-$$ making-$$
for input in folder/*; do
  if mv -- "$input" "$staging" 2>/dev/null; then
    bin/myProgram "$staging" >"$output"
    mv -- "$output" "$destination"
    mv -- "$staging" "$input"

A simple way to process the files in parallel using a widely-available tool is GNU make, using the -j flag for parallel execution. Here's a makefile for this task (remember to use tabs to indent commands):

all: $(patsubst folder/%,done/%,$(wildcard folder/*))
done/%: folder/%
        ./bin/myProgram $< >$@.tmp
        mv $@.tmp $@

Run make -j 3 to run 3 instances in parallel.

See also Four tasks in parallel... how do I do that?

| improve this answer | |
  • Thank you very much for this answer. If I could accept two answers, I would! – stefan May 11 '12 at 10:42

I have the feeling you are really trying to run multiple jobs in parallel and that the lock file is simply a means to an end.

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

parallel ./bin/myProgram ::: ./folder/*

It will run myProgram on each core in parallel.

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

| improve this answer | |

The problem with locking is that you need a method that creates a lock which is uninterruptable (sometimes called atomar). As Chris has wrote in his answer mkdir is such an uninterruptable operation (creating a file is no such operation).

There is also a high-level command - usally hidden in the procmail package: lockfile. That command has some nice features and can easily be used in your own scripts without the need to "reinvent the wheel" (for instance writing your own function that locks based on directory creation).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.