2

I found a script on GitHub which I've slightly modified to fit the needs of the program I'm trying to run in a queue.

It is not working however and I'm not sure why. It never actually echos the jobs to the queue file.

Here is a link to the GitHub page:

https://gist.github.com/tubaterry/c6ef393a39cfbc82e13b8716c60f7824

Here is the version I modified:

#!/bin/sh

END="END"
true > queue

tail -n+0 -f queue | parallel -j 16 -E "$END"

while read i; do
    echo "command [args] > ${i%.inp}.log 2> ${i%.inp}.err" > queue
done < "jobs.txt"

echo "$END" >> queue
echo "Waiting for jobs to complete"

while [ "$(pgrep 'perl /usr/local/bin/parallel' | grep -evc 'grep' | tr -d " ")" -gt "0" ]; do
    sleep 1
done

touch killtail
mv killtail queue
rm queue

The only thing I can think of is that one of these steps isn't operating as expected on OpenBSD. But I rearranged a step and everything executes without errors but it only submits one job. The change was moving tail -n+0 -f queue | parallel -j 16 -E "$END" after the first while loop and changing true > queue to touch queue since I'm not quite sure what true > queue means.

Any help would be appreciated.

EDIT:

I have a jobs.txt file filled with the path the the input files to the command I plan to run. The files in jobs.txt would be one of the arguments to command and then I output the results of the calculation to a log file and any errors to an error file.

My expectation is that each job will be added to queue and parallel will execute up to 16 jobs, one per core as one of the arguments to command is the utilisation of one core per calculation. This will continue until it reaches the “END” which is signified by the -E argument to parallel.

As written, nothing echos from jobs.txt to queue. I will try again with a >>

I have questioned quite a few things in the original script. I changed the things I’m sure about but some of the functionality I was very confused by and decided to leave it as is.

One of those things I’m not clear on is tail -n+0

I have no idea what that is doing

EDIT2:

${PROGRAM} ${JOB}.inp ${NCPU} > ${JOB}.log 2> ${JOB}.err

${JOB} is a reference to anywhere between 1 and ∞ calculations depending on how many I need to do at a given time. Currently, jobs.txt has 374 individual tests that I need to run. ${PROGRAM} is the software that takes the parameters from ${JOB}.inp and calculates accordingly. ${NCPU} is how many cores I want to use per job; currently I am trying to run each job in serial on a 16-core processor.

The goal is to queue as many calculations as I need to without ever typing that full command in. I just want to generate a list using find calculations -name '*.inp' -print > jobs.txt and then run a script such as SerialRun.sh or ParallelRun.sh and let it crank out results. The jobs may be nested in many different directories depending on how different users choose to organise their work and this method using find allows me to very quickly submit a job and generate results to the correct paths. As each calculation finishes, I can then analyse the data while the system continues to run through the tests.

The script very well may be over complicated. I was looking for a job queue system and found nqs which became the GNU Parallel project. I cannot find many examples of queueing jobs with parallel but came across that script on GitHub and decided to give it a shot. I have quite a few issues with how it is written but I don't understand parallel well enough to question it.

I figured it should be a bit simpler than this to build a queue for it.

EDIT3:

Maybe the correct way to go about this is to just do:

while read i; do
    command "$i" > "${i%.inp}".log 2> "${i%.inp}".err | parallel -j 16
done < "jobs.txt"

Would that work?

7
  • There are quite a few nonsensical steps in that script that suggest the author isn't particularly experienced (e.g. true > file can be written as > file and the last three commands (touch, mv and rm) are equivalent to just running rm queue directly), but my first guess is that you don't have a file called jobs.txt in the directory where you are running the script or that it exists and it is empty. Is that the case?
    – terdon
    Jun 23 at 15:30
  • Oh hang on, also, you are feeding overwriting the contents of queue in the whlie loop, did you mean to append with >> instead? Please edit your question and explain what you expect this script to do. We have no idea what is supposed to happen, what data you give it or what output you expect.
    – terdon
    Jun 23 at 15:36
  • I made an edit to the question
    – brokaryote
    Jun 23 at 16:35
  • But why aren't you just passing the file with jobs to parallel directly?
    – terdon
    Jun 23 at 18:19
  • And what is in the jobs.txt file? Your loop, if it worked, would just run "command [args] multiple times, saving the output and errors to different files but running the same command each time. Is that what you want? Please explain what you actually need to do here, the script is needlessly complicated and, as far as I can tell, not what you want at all. If you can show us a few examples from jobs.txt and the commands you are trying to generate from them, we can give you a simpler way.
    – terdon
    Jun 23 at 18:37

1 Answer 1

3

You don't need this complex script, parallel can do what want by itself. Just remove the .inp extension from your list of files using sed or any other tool of your choice, and feed the base name to parallel like this:

sed 's/\.inp//' jobs.txt | parallel -j 16 "${PROGRAM} {}.inp > {}.log 2> {}.err"

The {} notation is part of parallel's basic functionality, described in man parallel as follows:

{} Input line.

This replacement string will be replaced by a full line read from the input source. The input source is normally stdin (standard input), but can also be given with --arg-file, :::, or ::::.

So it is simply replaced by whatever you pass to parallel, in this case the list of file names with their extension removed by sed.

Alternatively, you can use {.} which is:

{.} Input line without extension.

This replacement string will be replaced by the input with the extension removed. If the input line contains . after the last /, the last . until the end of the string will be removed and {.} will be replaced with the remaining. E.g. foo.jpg becomes foo, subdir/foo.jpg becomes subdir/foo, sub.dir/foo.jpg becomes sub.dir/foo, sub.dir/bar remains sub.dir/bar. If the input line does not contain . it will remain unchanged.

The replacement string {.} can be changed with --extensionreplace

With this, you don't even need the jobs.txt file. If all of your files are in the same directory, you can do:

parallel -j 16 "${PROGRAM} {.}.inp > {.}.log 2> {.}.err" ::: *.inp

Or, to make it recursively descend into subdirectories, assuming you are using bash, you can do:

shopt -s globstar
parallel -j 16 "${PROGRAM} {.}.inp > {.}.log 2> {.}.err" ::: **/*.inp
4
  • Thank you for the help on this. This is much simpler and makes much more sense now. If I could vote up your answer I would but I don’t have enough points on is stack yet
    – brokaryote
    Jun 24 at 0:49
  • @brokaryote don't worry about it, I have more than enough points already! I'm glad I could help.
    – terdon
    Jun 24 at 10:15
  • @OleTange huh, you're absolutely right, unsurprisingly enough. I could have sworn I managed to break it with a file with spaces, but I just tried touch "a bad name"; echo * | parallel "cp {} ../bars 2>{}.err" and everything worked like a charm. A newline breaks it, but that isn't relevant here since we're cating a text file, and in any case quoting doesn't fix that. Thanks!
    – terdon
    Jun 27 at 16:30
  • If you want newline to work: -0 (and your input then needs to be \0 terminated. Slightly non-intuitively you also need -0 if you use ::: with values containing newline.
    – Ole Tange
    Jun 27 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.