I have the following in a shell script:

for file in $local_dir/myfile.log.*; 
        file_name=$(basename $file); 
        server_name=$(echo $file_name | cut -f 3 -d '.');
        file_location=$(echo $file);

        mv $file_location $local_dir/in_progress1.log

        mysql -hxxx -P3306 -uxxx -pxxx -e "set @server_name='${server_name}'; source ${sql_script};"

        rm $local_dir/in_progress1.log

It basically gets all files in a directory that match the criteria, extracts a servername from the filename, before passing it across to a MySQL script for procesing.

What I am wondering is if I have 10 files that take 60 seconds each to complete, and after 5 minutes I then start a second instance of the shell script:

  • a) will the second script still see the files that havent been processed
  • b) will it cause problems for the first instance if it deletes files

or will I be able to run them in parallel without issue?


One would assume that "60 seconds" (and even "5 minutes") is just a good estimate, and that there is a risk that the first batch is still in progress when the second batch is started. If you want to separate the batches (and if there is no problem aside from the log-files in an occasional overlap), a better approach would be to make a batch number as part of the in-progress filenaming convention.

Something like this:

[[ -s ]] $local_dir/batch || echo 0 > $local_dir/batch
batch=$(echo $local_dir/batch)
expr $batch + 1 >$local_dir/batch

before the for-loop, and then at the start of the loop, check that your pattern matches an actual file

[[ -f "$file" ]] || continue

and use the batch number in the filename:

mv $file_location $local_dir/in_progress$batch.log

and for forth. That reduces the risk of collision.

  • OK cheers, some files will take a second, others could take 20 minutes, so I picked 60 seconds as an average. It sounds similar to what I was thinking, where I renamed the files inside the loop in_progress1.log for script 1 and in_progress2.log` for script2 etc. I'll see if I can make it work. – IGGt Nov 5 '15 at 16:35

There is an answer above that provides some good solutions to the problem, but I thought I would provide a bit of explanation on the why of what the problem is.

For the most part: as long as your renamed log files (the in progress ones) do not meet the criteria, you are probably safe to run this with minimal risk. You will still get some errors though...

Your list of files is generated on script run. So what would end up happening is that:

Script A gets a list of 10 files. Begins processing, 5 files in (5 remaining) script B gets a list of 5 remaining files, begins processing. Script a then goes to process next file on its list(which is the same as the file script B has started to process) it will error because the file has been renamed. So with error handling, this it could theoretically move on to the next in its list and function without a problem. But, obviously there is ALWAYS the chance that the stars align, but scripts hit the same file at the same time, and something unexpected happens. Weigh that risk as you will.

A potentially more elegant solution would be to convert this to a python script and look into parallel for loops which would allow you to create a single for loop, and run through it in parallel, allowing one script to do the work of two or more.

  • cheers, I was wondering if the list was generated on runtime or after each file was picked up, You explained it really well. I like the idea of the Python script, alas it is one of the few languages I've not spent much time with for some reason. – IGGt Nov 5 '15 at 16:38

Another way to do it is to implement a simple batch queue in your script.

At the start of the script, you could do something like this:

mkdir -p $localdir/batch

# get list of current log files
find $local_dir/ -name 'myfile.log.*' > "$BATCHTMP"

# exclude any log files already in other batches
grep -vF -f <(sort -u $localdir/batch/batch.*) < "$BATCHTMP" > "$MYBATCH"

rm -f "$BATCHTMP"

# only process log files that are in my batch
for lf in $(cat "$MYBATCH") ; do
# somewhere in here, mv or rm the logfile being processed
# so it doesn't get processed again in a later batch run

rm -f "$MYBATCH"

This is, of course, only a simple outline of what needs to be done.

BTW, this could also be done in a wrapper script that does nothing but generate the batch file and then run the main script.

  • cheers, Looking at this When it runs the section BATCHTMP=$(mktemp batch) and BATCHTMP2 it gives the error mktemp: too few X's in template batch' – IGGt Nov 6 '15 at 16:23
  • @IGGt, fixed. added .XXXXXXXXXX to the mktemp, also removed unnecessary $BATCHTMP2 – cas Nov 6 '15 at 21:54
  • cheers, (sorry,Ive been away), when I run this $MYBATCH is always /batch/batch.24500. Should this be different each time? On the line grep -vF. . . . I then just get permission denied errors every time. Even with sudo. – IGGt Nov 13 '15 at 14:31
  • $$ is the PID of the sh/bash process that runs the script, so it's supposed to be different on every run. re: permissions, does the bigger script set the umask to something unusual, without read permission? – cas Nov 13 '15 at 15:12
  • OK, the $$ makes sense now. I checked permissions on $local_dir/batch and it looks fine (set to 777 to make sure), also my test myfile.* looked normal, but I set them to 777 as well, but still get the same error -bash: /batch/batch.1400: Permission denied followed by sort: open failed: /batch/batch.*: No such file or directory – IGGt Nov 13 '15 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.