Can I for example execute:
parallel -j 200 < list0
Where "list" has:
nice -n -20 parallel -j 100 < list2 nice -n -20 parallel -j 100 < list1
Would this be feasible/possible?
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
Not only is it possible; it is also recommended in some situations.
GNU Parallel takes around 10 ms to run a job. So if you have 8 cores and the jobs you run take less than 70 ms, then you will see GNU Parallel use 100% of a single core, and yet there will be idle time on other cores. Thus you will not use 100% of all cores.
The other situation where it is recommended is if you want to run more jobs than
-j0 will do. Currently
-j0 will run around 250 jobs in parallel unless you adjust some system limits. It makes perfect sense to run more than 250 jobs if the jobs are not limited by CPU and disk I/O. This is for example true if network latency is the limiting factor.
However, using 2 lists is not the recommended way to split up jobs. The recommended way is to use GNU Parallel to call GNU Parallel:
cat list0 | parallel -j20 --pipe parallel -j100
That will run 2000 jobs in parallel. To run more adjust
-j. It is recommended that the outer (the 20) is at least the number of cores, so that there will be at least one GNU Parallel process on each core.
Using this technique you should have no problem starting 20000 jobs in parallel; when you get over 32000 processes things start acting up.
By first running:
echo 4194304 | sudo tee /proc/sys/kernel/pid_max
I was able to run:
seq 1000000 2000000000 | parallel -j16 --roundrobin --pipe parallel -j0 --pipe parallel -j0 sleep
which will start 1 million processes in parallel (it takes 300 G RAM on my system).
I don't see why it would not be possible -- the system can certainly juggle 200 parallel tasks.
However, it almost certainly is not desirable, unless there is some specific reason you need that exact number of tasks running in parallel. This seems unlikely; the only reason I could see would be because you need them all existing at the same time because they need to exchange information, or exchange information with something else in a chaotic and indeterminate way (e.g. for testing a server program).
The reason it is not otherwise desirable is because the ideal state, in terms of efficiency, is for the system to run a number of processes equal to the number of available processor cores. Since processes to some extent often involve bottlenecks outside the CPU -- e.g. disk I/O -- this generallized ideal number ranges as a matter of opinion from the number of cores + 1 up to the number of cores * 2.
The reason this is the ideal state efficiency wise is that if a task itself consumes 1 million units of processor time, running the same task 10 times sequentially will consume 10 million units and running the same task in parallel will consume 10 million units. However, in the latter case, if there are fewer than 10 CPU's, there is an additional cost because the system must constantly switch back and forth from one task to another.
This is also why generally a system with 2 x 2 Ghz cores is faster than a system with 4 x 1 Ghz cores. The primary reason for the evolution of multi-core systems is because it becomes increasingly difficult to manufacture increasingly fast CPUs, and beyond a certain relatively low point, it is impossible. Hence the solution is to manufacture systems with more processor cores.
In short, if you need to do 20 things as quickly as possible and you have 4 cores, the fastest way to do this is to do them in 5 sets of 4, or 4 sets of 5 to allow for the idle time of waiting on I/O.
parallel allows you to feed it a list of indefinite length yet limit the number of jobs running at once (and note the default for this number is the number of cores).
There is a sort of exception to this, although it usually relates to certain kinds of singular multi-threaded programs (i.e., not a bunch of separate programs, but one program which occupies multiple cores). This is because when a program can get something done by doing it with relatively independent branches that only need to coordinate occasionally ("occasionally" might still be as frequent as 10 or 20 times per second), it is much easier, and often more flexible, to design the program to do this in independent threads than to design it to cycle the tasks in an arbitrary (asychronous) manner. Graphically intense and/or interactive programs such as video games and CAD systems fall into this category.