1

Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of which at most L < M can be running concurrently.

Now, suppose that I want to run N independent commands as quickly as possible.

If N ≤ M, I can just submit each command as a separate job to the scheduler, and be done with.

But what if N > M? Or N ≫ M even?


The N ≫ M scenario occurs extremely often in my line of work, so often in fact that a hope to find tools to facilitate dealing with it would not be unreasonable1.

One very general and straightforward way to get around the scheduler-imposed limits is to split the N independent commands into L separate one-time "batching" scripts, and submit the latter to the scheduler, as L separate jobs2.

Granted, creating such one-time batching scripts is a dull, somewhat annoying chore, but someone who is handy with their shell, or with a scripting language like Python, Perl, etc., can easily take care of it, and even home-roll their own hacks to automate it.

My question is, however, are there publicly (and freely) available tools in the Unix ecosystem that can be used even by those with less programming skill to automate the chore of generating L such batching scripts, given as input a list of N independent commands?


1Actually, the scenario occurs so often that I am surprised that schedulers do not already have built-in support for it. At least the schedulers that I am most familiar with (Slurm and LSF) do not have any such support, as far as I can tell. Please correct me if I missed something.

2 More generally, one could to batch the N commands into k batching scripts, as long as k ≤ M, but, in my experience, choosing k = L is the most straightforward way to achieve a maximal, or near-maximal, throughput under these constraints. The reasons for this are not too difficult to see, but an adequate discussion of the matter would require more time than I want to take up here.

1

1 Answer 1

1
L=10000
M=100000
N=10000000

jobs=$(wc -l < all_jobs.txt)
jobs_per_jobslot=$(( jobs/L + 1))

cat all_jobs.txt | parallel --pipe -N$jobs_per_jobslot 'cat > jobs_{#}'
# submit jobs_*

Spend 20 minutes reading chapter 1+2 of https://doi.org/10.5281/zenodo.1146014 Your command line will love you for it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .