Tag Info

Hot answers tagged

30

If you have a copy of xargs that supports parallel execution with -P, you can simply do printf '%s\0' *.png | xargs -0 -I {} -P 4 ./pngout -s0 {} R{} For other ideas, the Wooledge Bash wiki has a section in the Process Management article describing exactly what you want.


15

Under Linux, execute the sched_setaffinity system call. The affinity of a process is the set of processors on which it can run. There's a standard shell wrapper: taskset. For example, to pin a process to CPU #0 (you need to choose a specific CPU): taskset -c 0 mycommand --option # start a command with the given affinity taskset -c -p 0 1234 # ...


13

Most software build processes use make. Make sure you make make use the -j argument with a number usually about twice the number of CPUs you have, so make -j 8 would be appropriate for your case.


12

You have been hit by the confusion with Tollef's parallel from moreutils. See https://www.gnu.org/software/parallel/history.html You can install GNU Parallel simply by: wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel chmod 755 parallel cp parallel sem Watch the intro videos for GNU Parallel to learn more: ...


11

Actually, you don't have a problem with make, but with your command: tex dummy.tex &> /dev/null; Runs 'tex' in the background. You don't need to remove '>/dev/null', but '&' is sending 'tex' to the background. Try this, it must be fine for you: tex dummy.tex > /dev/null; or run everything in the same subshell, like this: (tex dummy.tex ...


11

Using GNU Parallel, $ parallel -j ${jobs} wget < urls.txt or xargs from GNU Findutils, $ xargs -P ${jobs} wget < urls.txt where ${jobs} is the maximum number of wget you want to allow to run concurrently. Without -j/-P, parallel will run as many jobs at a time as CPU cores (which doesn't necessarily make sense for wget bound by network IO), ...


11

for stuff in things do ( something with stuff ) & done wait # for all the something with stuff Whether it actually works depends on your commands; I'm not familiar with them. The rm *.mat looks a bit prone to conflicts if it runs in parallel...


11

Use wait. For example: Data1 ... > Data1Res.csv & Data2 ... > Data2Res.csv & wait AnalysisProg will: run the Data1 and Data2 pipes as background jobs wait for them both to finish run AnalysisProg. See, e.g., this question.


10

You can't spawn threads from a shell. You don't want to write to the same file from multiple processes. If all your random program does is generate a single number, it should be fast enough that your loop will be io bound. if you can, you should edit it to take an argument and print that many numbers. if the actual execution is the bottleneck, you ...


10

Reniceing the process group to -20 is a bad idea. This niceness level should be used only by the top-priority system-critical tasks. Otherwise you can loose responsiveness or even freeze the system. And the potential compilation-time benefit would be marginal. Apart from what Caleb already suggested, if you compile a lot, you can also speed up builds using ...


10

#!/bin/bash # set -x # debug version N=${1:-123} n=${2:-45} workers=${workers:-${3:-10}} ((workers < 1)) && ((workers = 1)) ((workers > 20)) && ((workers = 20)) ((min=100000000000000)) #set min to some garbage value work() { for i in ${*}; do for (( j=1; j<=${n}; j++ )); do val=$(/path/to/a.out) val2=$(echo ...


10

A problem with split --filter is that the output can be mixed up, so you get half a line from process 1 followed by half a line from process 2. GNU Parallel guarantees there will be no mixup. So assume you want to do: A | B | C But that B is terribly slow, and thus you want to parallelize that. Then you can do: A | parallel --pipe B | C GNU Parallel ...


9

Will the stdout from different programs be messed when running in parallel? Not if they are all independent processes writing to separate files, which they appear to be -- each instance of B is distinct, and outputs to its own place.


8

GNU Parallel is specifcally designed to solve this problem: echo -n $IPs | parallel -d ' ' -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string' If your IPs are in a file it is even prettier: cat IPs | parallel -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string' To learn more watch the intro video: ...


8

When you write A | B, both processes already run in parallel. If you see them as using only one core, that's probably because either of CPU affinity settings (perhaps there is some tool to spawn a process with different affinity) or because one process isn't enough to hold a whole core, and the system "prefers" not to spread out computing. To run several ...


8

Putting multiple jobs in the background is a good way of using the multiple cores of a single machine. parallel however, allows you to spread jobs across multiple servers of your network. From man parallel: GNU parallel is a shell tool for executing jobs in parallel using one or more computers. The typical input is a list of files, a list of ...


7

Using GNU Parallel: #!/bin/bash N=$1 n=$2 arr=($( # Generate all combinations of 1..n and 1..N parallel -k --tag /path/to/a.out {1} {2} '|' bc :::: <(seq $N) <(seq $n) | perl -ane 'BEGIN{$min=1e30} $last||=$F[0]; if($F[0] != $last) {print $min,"\n";$min=1e30;$last=$F[0]} $min = $F[2]<$min ? $F[2] : $min; END {print $min,"\n"}' )) echo ${arr[*]} ...


7

In addition to solutions already proposed, you can create a makefile that describes how to make a compressed file from uncompressed, and use make -j 4 to run 4 jobs in parallel. The problem is that you will need to name compressed and uncompressed files differently, or store them in different directories, else writing a reasonable make rule will be ...


7

(I would make this a comment, but I lack the privileges.) You can also probably just use parallel --gnu. You said that Ubuntu 12(.04?) uses Tollef's version, but it's actually the gnu implementation, it just defaults to Tollef's syntax by default for some reason. I added alias parallel='parallel --gnu' to my login script so I don't have to fuss with it ...


7

Why don't you just fork (aka. background) them? foo () { local run=$1 fsl5.0-flirt -in $kar"deformed.nii.gz" -ref normtemp.nii.gz -omat $run".norm1.mat" -bins 256 -cost corratio -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12 fsl5.0-flirt -in $run".poststats.nii.gz" -ref $kar"deformed.nii.gz" -omat $run".norm2.mat" -bins 256 -cost ...


7

A classic case of RTFM (all of it!). The -T option to GNU tar will read the files to be archived from another file (in my case, /dev/stdin, you can also use -), and there's even a --remove-files option: alias magic_otf_compressor='tar --create -T - --remove-files -O | pixz' (using the parallel version of xz for compression, but you can use your preferred ...


7

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel: ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > ...


6

If you want to parallelize on a machine with multiple cores, you can just use (GNU) xargs, e.g.: echo seq_[0-9][0-9].gz | xargs -n 1 -P 16 ./crunching Meaning: xargs starts up to 16 processes in parallel of ./crunching using 1 token from stdin for each process. You can also use split in combination with xargs. Or you can create a simple Makefile for Job ...


6

In this specific case, you're passing -P 3 to xargs. -P max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n op‐ tion with -P; otherwise chances are that only one exec will be done. Because you're running them in ...


5

GNU Parallel does that and more (using ssh). It can even deal with mixed speed of machines, as it simply has a queue of jobs, that are started on the list of machines (e.g. one per CPU core). When one jobs finishes another one is started. So it does not divide the jobs into clusters before starting, but does it dynamically. Watch the intro videos to learn ...


5

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this: parallel ./pngout -s0 {} R{} ::: *.png You can install GNU Parallel simply by: wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel chmod 755 parallel cp parallel sem Watch the intro videos for GNU Parallel to learn more: ...


5

To list all files start with number in a directory, find . -maxdepth 1 -regextype "posix-egrep" -regex '.*/[0-9]+.*\.mp3' -type f Problem with your approach is that the find returns a relative path of a file and you are just expecting a filename itself.


5

Simply remove the ; character, so in final : for i in *; do something.py $i & done And for running N instance of your script at the same time, see man 1 parallel See http://www.gnu.org/software/parallel/


5

As mavillan already suggested, just use terminator. It allows to display many terminals in a tiled way. When enabling the broadcasting feature, you can enter the very same command simultaneously on each terminal. Here is an example with the date command broadcasted to a grid of 32 terminals.


5

lftp would do this with the command mirror -R -P 20 localpath - mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once. As explained in man lftp: mirror [OPTS] [source [target]] Mirror specified source directory to local target directory. If target directory ends with a ...



Only top voted, non community-wiki answers of a minimum length are eligible