1

I would like to run a simple python script in ~6500 directories. The easiest, and least efficient way is to do:

for d in *_directorynumber; do (cd "$d" && cp ../script.py . && python ./script.py );done

This obviously takes forever. Instead, I try to run in parallel:

task(){
   cd "$d" && python ./script.py . 
}

Then run this task as such:

for d in *_directorynumber; do 
  task "$d" &
done

After ~500 runs or so, I get the following error:

-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes

Is there another way to parallelize?

4
  • 3
    parallel or xargs -P (Depending on versions etc.) see e.g. the questions here: unix.stackexchange.com/questions/tagged/gnu-parallel
    – ilkkachu
    Mar 24, 2021 at 13:52
  • You are running too many processes. Perhaps you need to increase nproc, but it is more likely that you just have to increase the user's ulimit. Mar 24, 2021 at 13:54
  • 3
    Why would you copy the script each time? Is that necessary for some reason? Surely a symlnk would be enough even if the script required the data to be in the same directory.
    – terdon
    Mar 24, 2021 at 14:00
  • Wouldn't it be better to have your script picking the folders, and running stuff in parallel inside a single script?
    – Panki
    Mar 24, 2021 at 14:30

2 Answers 2

3

There is a limit on the number of processes. You can display it with the command ulimit -u. You may be able to increase it with the same commands. If it is a shared machine, you probably don't have the privileges to do this.

Running 6500 processes in parallel is most likely a bad idea.

  • You need RAM for 6500 processes.
  • If your task is CPU bound, it will be slower, because it needs much more context switches.
  • If your task is I/O bound, having 6500 processes access different directories will also make it slower.

Your task may benefit from some parallel processes, but you should limit it to a small multiple of the number of CPU cores you have.

1
  • Linux is pretty efficient at context switches. I just tested running 1000 procs in parallel on a CPU with 4 CPU threads. The difference to running 4 procs in parallel was minimal (< 1%). The other arguments are valid, though.
    – Ole Tange
    Mar 27, 2021 at 3:38
0

Try:

parallel 'cd {} && cp ../script.py . && python ./script.py' :::  *_directorynumber

This will run one script.py per CPU thread until all are done.

If script.py is not CPU bound you can adjust the number of jobs with --jobs:

  • --jobs 10 run exactly 10 in parallel
  • --jobs 200% run 2x CPU threads in parallel
  • --jobs 0 run as many as you can until you hit a limit (such as: -bash: fork: retry: No child processes) then stay below that limit.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .