5

Can I for example execute:

parallel -j 200 < list0

Where "list" has:

nice -n -20 parallel -j 100 < list2
nice -n -20 parallel -j 100 < list1

Would this be feasible/possible?

  • have you tried it ? – Kiwy Apr 10 '14 at 5:54
  • why are you adding nice -n -20 to the mix? just trying to wreck the machine? Also, you have to run as root to get nice levels lower than 0 – pqnet Aug 17 '14 at 19:20
7

Not only is it possible; it is also recommended in some situations.

GNU Parallel takes around 10 ms to run a job. So if you have 8 cores and the jobs you run take less than 70 ms, then you will see GNU Parallel use 100% of a single core, and yet there will be idle time on other cores. Thus you will not use 100% of all cores.

The other situation where it is recommended is if you want to run more jobs than -j0 will do. Currently -j0 will run around 250 jobs in parallel unless you adjust some system limits. It makes perfect sense to run more than 250 jobs if the jobs are not limited by CPU and disk I/O. This is for example true if network latency is the limiting factor.

However, using 2 lists is not the recommended way to split up jobs. The recommended way is to use GNU Parallel to call GNU Parallel:

cat list0 | parallel -j20 --pipe parallel -j100

That will run 2000 jobs in parallel. To run more adjust -j. It is recommended that the outer (the 20) is at least the number of cores, so that there will be at least one GNU Parallel process on each core.

Using this technique you should have no problem starting 20000 jobs in parallel; when you get over 32000 processes things start acting up.

2

I don't see why it would not be possible -- the system can certainly juggle 200 parallel tasks.

However, it almost certainly is not desirable, unless there is some specific reason you need that exact number of tasks running in parallel. This seems unlikely; the only reason I could see would be because you need them all existing at the same time because they need to exchange information, or exchange information with something else in a chaotic and indeterminate way (e.g. for testing a server program).

The reason it is not otherwise desirable is because the ideal state, in terms of efficiency, is for the system to run a number of processes equal to the number of available processor cores. Since processes to some extent often involve bottlenecks outside the CPU -- e.g. disk I/O -- this generallized ideal number ranges as a matter of opinion from the number of cores + 1 up to the number of cores * 2.

The reason this is the ideal state efficiency wise is that if a task itself consumes 1 million units of processor time, running the same task 10 times sequentially will consume 10 million units and running the same task in parallel will consume 10 million units. However, in the latter case, if there are fewer than 10 CPU's, there is an additional cost because the system must constantly switch back and forth from one task to another.

This is also why generally a system with 2 x 2 Ghz cores is faster than a system with 4 x 1 Ghz cores. The primary reason for the evolution of multi-core systems is because it becomes increasingly difficult to manufacture increasingly fast CPUs, and beyond a certain relatively low point, it is impossible. Hence the solution is to manufacture systems with more processor cores.

In short, if you need to do 20 things as quickly as possible and you have 4 cores, the fastest way to do this is to do them in 5 sets of 4, or 4 sets of 5 to allow for the idle time of waiting on I/O. parallel allows you to feed it a list of indefinite length yet limit the number of jobs running at once (and note the default for this number is the number of cores).

There is a sort of exception to this, although it usually relates to certain kinds of singular multi-threaded programs (i.e., not a bunch of separate programs, but one program which occupies multiple cores). This is because when a program can get something done by doing it with relatively independent branches that only need to coordinate occasionally ("occasionally" might still be as frequent as 10 or 20 times per second), it is much easier, and often more flexible, to design the program to do this in independent threads than to design it to cycle the tasks in an arbitrary (asychronous) manner. Graphically intense and/or interactive programs such as video games and CAD systems fall into this category.

  • Well I have to download 3 million files with wget and I'm noticing that the CPU/Network/IO is not being taxed at all, this is why I'm trying to increase the number of processes to see where it goes. – Dominique Apr 10 '14 at 6:08
  • So you are telling me that it would be faster to run 8 wget processes simultaneously(I have 8 threads) that to run more? – Dominique Apr 10 '14 at 6:09
  • 3
    @Dominique maybe the whole thing is waiting for the servers you wget from to respond? If that is the case, your CPUs and the rest of your system are unlikely to get taxed. – Anthon Apr 10 '14 at 6:22
  • "The CPU/Network/IO" is not being taxed... -> These are two or three different things. In the case of downloading, you have, in order of the significance of the bottleneck, 1) Network I/O, 2) Disk I/O, presuming you are saving to disk, and 3) CPU time. In other words, in most cases the CPU will have to wait on disk I/O and disk I/O will have to wait on network I/O. So in theory doing this in parallel will probably not make any difference at all. However, waiting on the network involves variables outside the system independent to each download, so doing many of them at the same time: – goldilocks Apr 10 '14 at 6:26
  • A) Should be faster since the faster downloads won't have to wait for the slower ones, B) Certainly won't incurr what I called the "additional cost" of parallel execution because the CPU will be relatively idle in any case. All it will do is occupy more RAM, but in the case of wget that won't be very significant. To be clear: don't expect this to occupy much CPU time no matter how many downloads you execute simultaneously. It simply is not a processor instensive activity. – goldilocks Apr 10 '14 at 6:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.