Hot answers tagged

41

Under Linux, execute the sched_setaffinity system call. The affinity of a process is the set of processors on which it can run. There's a standard shell wrapper: taskset. For example, to pin a process to CPU #0 (you need to choose a specific CPU): taskset -c 0 mycommand --option # start a command with the given affinity taskset -c -p 0 1234 # ...


32

If you have a copy of xargs that supports parallel execution with -P, you can simply do printf '%s\0' *.png | xargs -0 -I {} -P 4 ./pngout -s0 {} R{} For other ideas, the Wooledge Bash wiki has a section in the Process Management article describing exactly what you want.


29

for stuff in things do ( something with stuff ) & done wait # for all the something with stuff Whether it actually works depends on your commands; I'm not familiar with them. The rm *.mat looks a bit prone to conflicts if it runs in parallel...


24

Putting multiple jobs in the background is a good way of using the multiple cores of a single machine. parallel however, allows you to spread jobs across multiple servers of your network. From man parallel: GNU parallel is a shell tool for executing jobs in parallel using one or more computers. The typical input is a list of files, a list of ...


23

Why don't you just fork (aka. background) them? foo () { local run=$1 fsl5.0-flirt -in $kar"deformed.nii.gz" -ref normtemp.nii.gz -omat $run".norm1.mat" -bins 256 -cost corratio -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12 fsl5.0-flirt -in $run".poststats.nii.gz" -ref $kar"deformed.nii.gz" -omat $run".norm2.mat" -bins 256 -cost ...


20

Using GNU Parallel, $ parallel -j ${jobs} wget < urls.txt or xargs from GNU Findutils, $ xargs -n 1 -P ${jobs} wget < urls.txt where ${jobs} is the maximum number of wget you want to allow to run concurrently (setting -n to 1 to get one wget invocation per line in urls.txt). Without -j/-P, parallel will run as many jobs at a time as CPU cores (...


18

As mavillan already suggested, just use terminator. It allows to display many terminals in a tiled way. When enabling the broadcasting feature, you can enter the very same command simultaneously on each terminal. Here is an example with the date command broadcasted to a grid of 32 terminals.


18

Use wait. For example: Data1 ... > Data1Res.csv & Data2 ... > Data2Res.csv & wait AnalysisProg will: run the Data1 and Data2 pipes as background jobs wait for them both to finish run AnalysisProg. See, e.g., this question.


17

I'm not entirely sure what you're asking here. Yes, top shows CPU usage as a percentage of a single CPU by default. That's why you can have percentages that are >100. On a system with 4 cores, you can see up to 400% CPU usage. You can change this behavior by pressing I (that's Shift + i and toggles "Irix mode") while top is running. That will cause it to ...


13

Most software build processes use make. Make sure you make make use the -j argument with a number usually about twice the number of CPUs you have, so make -j 8 would be appropriate for your case.


11

You can't spawn threads from a shell. You don't want to write to the same file from multiple processes. If all your random program does is generate a single number, it should be fast enough that your loop will be io bound. if you can, you should edit it to take an argument and print that many numbers. if the actual execution is the bottleneck, you ...


11

When you write A | B, both processes already run in parallel. If you see them as using only one core, that's probably because either of CPU affinity settings (perhaps there is some tool to spawn a process with different affinity) or because one process isn't enough to hold a whole core, and the system "prefers" not to spread out computing. To run several B'...


11

A problem with split --filter is that the output can be mixed up, so you get half a line from process 1 followed by half a line from process 2. GNU Parallel guarantees there will be no mixup. So assume you want to do: A | B | C But that B is terribly slow, and thus you want to parallelize that. Then you can do: A | parallel --pipe B | C GNU Parallel ...


11

The words “CPU”, “processor” and “core” are used in somewhat confusing ways. They refer to the processor architecture. A core is the smallest independent unit that implements a general-purpose processor; a processor is an assemblage of cores (on some ARM systems, a processor is an assemblage of clusters which themselves are assemblages of cores). A chip can ...


11

There are two easy solutions for this. Basically, using xargs or parallel. xargs Approach: You can use xargs with find as follows: find . -type f -print0 | xargs -0 -P number_of_processes grep mypattern > output Where you will replace number_of_processes by the maximum number of processes you want to be launched. However, this is not guaranteed to ...


10

GNU Parallel is designed for this kind of tasks: parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output ::: *.input or: ls | parallel customScript -c 33 -I -file {} -a -v 55 '>' {.}.output It will run one jobs per CPU core. You can install GNU Parallel simply by: wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel ...


10

Reniceing the process group to -20 is a bad idea. This niceness level should be used only by the top-priority system-critical tasks. Otherwise you can loose responsiveness or even freeze the system. And the potential compilation-time benefit would be marginal. Apart from what Caleb already suggested, if you compile a lot, you can also speed up builds using ...


10

#!/bin/bash # set -x # debug version N=${1:-123} n=${2:-45} workers=${workers:-${3:-10}} ((workers < 1)) && ((workers = 1)) ((workers > 20)) && ((workers = 20)) ((min=100000000000000)) #set min to some garbage value work() { for i in ${*}; do for (( j=1; j<=${n}; j++ )); do val=$(/path/to/a.out) val2=$(echo ${val}...


10

for stuff in things do sem -j+0 ( something with stuff ) done sem --wait This will use semaphores, parallelizing as many iterations as the number of available cores (-j +0 means you will parallelize N+0 jobs, where N is the number of available cores). sem --wait tells to wait until all the iterations in the for loop have terminated execution before ...


10

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel: ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res....


9

Will the stdout from different programs be messed when running in parallel? Not if they are all independent processes writing to separate files, which they appear to be -- each instance of B is distinct, and outputs to its own place.


8

Using GNU Parallel: #!/bin/bash N=$1 n=$2 arr=($( # Generate all combinations of 1..n and 1..N parallel -k --tag /path/to/a.out {1} {2} '|' bc :::: <(seq $N) <(seq $n) | perl -ane 'BEGIN{$min=1e30} $last||=$F[0]; if($F[0] != $last) {print $min,"\n";$min=1e30;$last=$F[0]} $min = $F[2]<$min ? $F[2] : $min; END {print $min,"\n"}' )) echo ${arr[*]} ...


8

Just answering your first question. In the output of cat /proc/cpuinfo you can see the following information:- physical id : 0 siblings : 4 core id : 0 cpu cores : 2 You can see the count of siblings is 4 and cpu cores is 2. cpu cores being 2 is that total number of cores in the processor which can be checked from the spec given in the intel's ...


7

In addition to solutions already proposed, you can create a makefile that describes how to make a compressed file from uncompressed, and use make -j 4 to run 4 jobs in parallel. The problem is that you will need to name compressed and uncompressed files differently, or store them in different directories, else writing a reasonable make rule will be ...


7

Sample task task(){ sleep 0.5; echo "$1"; } Sequential runs for thing in a b c d e f g; do task "$thing" done Parallel runs for thing in a b c d e f g; do task "$thing" & done Parallel runs in N-process batches N=4 ( for thing in a b c d e f g; do ((i=i%N)); ((i++==0)) && wait task "$thing" & done )


7

A classic case of RTFM (all of it!). The -T option to GNU tar will read the files to be archived from another file (in my case, /dev/stdin, you can also use -), and there's even a --remove-files option: alias magic_otf_compressor='tar --create -T - --remove-files -O | pixz' (using the parallel version of xz for compression, but you can use your preferred ...


7

In your example the combine command will just be run as soon as the subshell exits (and provided the last background process was started without an error). The subshell will exit immediately after the jobs are started since there is no wait command. If you want to execute a command based on the return value of two or more simultaneous background processes, ...


6

You can build cpu-sets on the command line as well. man cpuset Later on you can assign (running) processes to these.


6

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this: parallel ./pngout -s0 {} R{} ::: *.png You can install GNU Parallel simply by: wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel chmod 755 parallel cp parallel sem Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/...


6

Simply remove the ; character, so in final : for i in *; do something.py $i & done And for running N instance of your script at the same time, see man 1 parallel See http://www.gnu.org/software/parallel/



Only top voted, non community-wiki answers of a minimum length are eligible