As part of my research project I'm processing huge amount of data splitted up into many files.

All files in folder foo have to be processed by the script myScript involving all elements of folder bar.

This is myScript:

for f in bar/*
    awk 'NR==FNR{a[$0]=$0;next}!a[$0]' $f $1 > tmp
    cp tmp $1

The first idea to just process all files with a for loop is valid:

for f in foo/*
    ./myScript $f

However, this simply takes forever. Simply starting every myScript in background by appending & would create thousands of parallely executed instances of awk and cp with huge input, which is obviously bad.

I thought of limiting the number of "threads" created with the following

for f in foo/*
    THREAD_COUNT=$(ps | wc -f)
    while [ $THREAD_COUNT -ge 12 ]
        sleep 1
        THREAD_COUNT=$(ps | wc -f)
    ./myScript $f &

As a side note: I'm comparing with 12, because I've got 8 cores on my nodes and apparently there's always bash, ps and wc running as well as the header line at the moment of the call of ps | wc -l.

Unfortunately the call of myScript causes more than one additional entry in ps, so the behaviour of my script wasn't as intended.

So here's my question: Is there a simpler way? A way which is more stable?

I'm not doing anything else on the nodes, so everything happening is caused by the scripts only.

  • Just a additional short question: I'm running this on nodes which I do NOT ssh into, so the script is running without me. Does this change the number of default active processes? – stefan Sep 6 '12 at 21:01
  • I assume you use a batch system to start your script. Have you looked at its documentation? Maybe it has the required functionality. – Ansgar Esztermann Sep 6 '12 at 21:23
  • it's a cluster using PBS, I didn't find something useful in that documentation. Since I need to preload some data, I think it wouldn't be a good idea to do it with PBS even if it's possible – stefan Sep 6 '12 at 21:24
  • 1
  • @jw013 Thanks for this link, I didn't find that one. Unfortunately I don't see an answer which suits my case. – stefan Sep 6 '12 at 21:37

While you can do this with a shell script, this is going at it the hard way. Shell scripts aren't very good at manipulating multiple background jobs.

My recommendation is to use GNU make or some other version of make that has a -j option to execute multiple jobs in parallel. Write each subtask as a makefile rule.

I think the makefile snippet below implements your rules, but your code was hard to follow so I might now have gotten it right. The first line enumerates the output files from the input files (note: never overwrite any input files! If the job stops in the middle for any reason, you'll end up with data for which you won't know whether it's been processed). The indented lines are the commands to run. Use a tab to indent each command, not 8 spaces. In these commands, $< represents the source file (a .in file), $@ represents the target (the .out file), and $* is the target without its extension. All $ signs in shell commands must be doubled, and each command line is executed in a separate subshell unless you put a \ at the end which cancels that newline (so the shell sees one long line starting with set -e and ending with done).

all: $(patsubst %.in,%.out,$(wildcard foo/*.in))
%.out: %.in
        cp $< $*.tmp.in
        set -e; \
        for f in bar/*; do \
          awk 'NR==FNR{a[$$0]=$$0;next}!a[$$0]' $$f $*.tmp.in >$*.tmp.out; \
          mv $*.tmp.out $*.tmp.in; \
        mv $*.tmp.in $@

Put this in a file called Makefile and call make -j12.

  • Thanks for your Makefile-approach. Actually since I do all the scripts with temporary files on a dedicated node, it's not a problem to overwrite input data. It's only copied back to my workspace if successful (and it's not overwritten there). I have additional logic (such as locking), but I didn't considered it necessary to add to the question. – stefan Sep 7 '12 at 7:13

Using GNU Parallel (http://www.gnu.org/software/parallel/) it looks like this:

parallel awk \'NR==FNR\{a\[\$0\]=\$0\;next\}\!a\[\$0\]\' {1} {2} '>{2}.tmp; mv {2}.tmp {2}' ::: bar/* ::: foo/*

This will run one job per core. Use -j150% to run 1.5 jobs per core.

If you just want to run several myScript in parallel do:

parallel ./myScript ::: foo/*

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1


You may try using ulimit. From bash man-page:

ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started  by  it, 
on systems  that  allow  such control.
-u     The maximum number of processes available to a single user

So if you put ulimit -u 8to an appropriate position inside the script, it limits the processes available to that shell to 8.

Did not test it, though.

  • Seems like a good and easy way. Unfortunately there is an error "fork: resource temporarily unavailable" now – stefan Sep 6 '12 at 21:39
  • That is expected - if you limit the maximum number of processes you will get an error on fork if you try to exceed that number. You can still accomplish what you want with a loop that constantly tries to spawn background jobs and constantly retries the ones that failed due to hitting the process limit. – jw013 Sep 6 '12 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.