3

I have hundreds of files to be processed but only want 10 processes to be running at a time. Suppose the "doSomething" process takes 20 seconds to complete. The following works but starts 10 processes almost simultaneously. ~20 seconds later the first set of 10 completes and the next set of 10 starts virtually simultaneously, and the series repeats. How can I stagger the starts to not occur simultaneuosly?

find ./someFiles* | xargs --max-args=1 --max-procs=10 ./doSomething

I would like the processes to start at least 2 seconds apart instead of 10 starting at virtually the same time.

8
  • Could you please describe what is the expected outcome, what do you want to happen? And maybe why? Of course, as long as the running processes are not 10, immediately new processes are raised. – thanasisp Nov 6 '20 at 15:57
  • I would like the processes to start at least 2 seconds apart instead of 10 starting at the virtually the same time. – gergNo Nov 6 '20 at 16:07
  • Care to give the duration of an individual ./doSomething task? Because a simplistic illustration of my idea would depend on it lasting some minimum time (say 10 or 20s) – A.B Nov 6 '20 at 16:28
  • You were asked to describe what is the expected outcome, what do you want to happen? And why? (this will help us come up with solutions that will solve your problem, may be not the solution that you are thinking of). However you just repeated what you said earlier (we have the ability to re-read the question, so this is not helpful). – ctrl-alt-delor Nov 6 '20 at 17:13
  • What is the problem with starting them at the same time. (I have some ideas, but it may not be your problem). – ctrl-alt-delor Nov 6 '20 at 17:15
1

So the reason for that is to avoid a spike in disk/network or other resources usage when many instances start together. At least for the first N of them you want a fixed interval of X seconds.

A simple workaround is to interpolate an additional xargs to delay the arguments. Like this:

find . -type f -print0 |
xargs -0 -n1 -P1 sh -c 'sleep 2; printf "%s\0" "$0"' |
xargs -0 -n1 -P4 sh do_something.sh

In the above I have N=4 and interval 2 seconds. For the first N arguments it will keep the interval. Then, in case of some executions ending closely, it can start executions closer in time, and this is what you request into this comment.

I have also assumed that time of executions will be not tiny, but several seconds or more. Also you can set a little higher delay if you still see spikes for the next executions. The main bottleneck at the beginning is avoided.


Testing

Here is some basic testing. The processing script, do_something.sh is taking random time from 10 to 20 seconds.

> cat do_something.sh 
printf "%s START processing %s\n" "$(date +"%H:%M:%S")" "$1"
sleep $(shuf -i10-20 -n1)
printf "%s END processing %s\n" "$(date +"%H:%M:%S")" "$1"

> touch file{1..10}

> find . -type f -name 'file*' -print0 |
> xargs -0 -n1 -P1 sh -c 'sleep 2; printf "%s\0" "$0"' |
> xargs -0 -n1 -P4 sh do_something.sh
02:03:22 START processing ./file6
02:03:24 START processing ./file9
02:03:26 START processing ./file8
02:03:28 START processing ./file2
02:03:38 END processing ./file8
02:03:38 START processing ./file7
02:03:40 END processing ./file6
02:03:40 START processing ./file1
02:03:41 END processing ./file9
02:03:41 START processing ./file3
02:03:45 END processing ./file2
02:03:45 START processing ./file4
02:03:55 END processing ./file3
02:03:55 END processing ./file7
02:03:55 START processing ./file10
02:03:55 START processing ./file5
02:04:00 END processing ./file1
02:04:02 END processing ./file4
02:04:05 END processing ./file10
02:04:13 END processing ./file5
0

Assuming it's a shell script of some kind, put this at the top:

export MYPID=$$
(
    flock 9
    mkdir -p .started-pids
    find .started-pids/ -type f ! -newermt '-2 seconds' -delete
    n=`find .started-pids/ -type f | wc -l`
    sleep $n
    sleep $n
    touch .started-pids/$MYPID
) 9> .lockfile

It's not quite accurate -- sometimes it will delay a bit more than it should, but not less, so your spike won't happen.

Change the lockfile and the pid count directory names to whatever you like, of course.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.