Tag Info

Hot answers tagged

29

If you have a copy of xargs that supports parallel execution with -P, you can simply do printf '%s\0' *.png | xargs -0 -I {} -P 4 ./pngout -s0 {} R{} For other ideas, the Wooledge Bash wiki has a section in the Process Management article describing exactly what you want.


12

Actually, you don't have a problem with make, but with your command: tex dummy.tex &> /dev/null; Runs 'tex' in the background. You don't need to remove '>/dev/null', but '&' is sending 'tex' to the background. Try this, it must be fine for you: tex dummy.tex > /dev/null; or run everything in the same subshell, like this: (tex dummy.tex ...


11

Under Linux, execute the sched_setaffinity system call. The affinity of a process is the set of processors on which it can run. There's a standard shell wrapper: taskset. For example, to pin a process to CPU #0 (you need to choose a specific CPU): taskset -c 0 mycommand --option # start a command with the given affinity taskset -c -p 0 1234 # ...


11

You have been hit by the confusion with Tollef's parallel from moreutils. See https://www.gnu.org/software/parallel/history.html You can install GNU Parallel simply by: wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel chmod 755 parallel cp parallel sem Watch the intro videos for GNU Parallel to learn more: ...


11

Use wait. For example: Data1 ... > Data1Res.csv & Data2 ... > Data2Res.csv & wait AnalysisProg will: run the Data1 and Data2 pipes as background jobs wait for them both to finish run AnalysisProg. See, e.g., this question.


10

#!/bin/bash # set -x # debug version N=${1:-123} n=${2:-45} workers=${workers:-${3:-10}} ((workers < 1)) && ((workers = 1)) ((workers > 20)) && ((workers = 20)) ((min=100000000000000)) #set min to some garbage value work() { for i in ${*}; do for (( j=1; j<=${n}; j++ )); do val=$(/path/to/a.out) val2=$(echo ...


10

You can't spawn threads from a shell. You don't want to write to the same file from multiple processes. If all your random program does is generate a single number, it should be fast enough that your loop will be io bound. if you can, you should edit it to take an argument and print that many numbers. if the actual execution is the bottleneck, you ...


10

Reniceing the process group to -20 is a bad idea. This niceness level should be used only by the top-priority system-critical tasks. Otherwise you can loose responsiveness or even freeze the system. And the potential compilation-time benefit would be marginal. Apart from what Caleb already suggested, if you compile a lot, you can also speed up builds using ...


9

A problem with split --filter is that the output can be mixed up, so you get half a line from process 1 followed by half a line from process 2. GNU Parallel guarantees there will be no mixup. So assume you want to do: A | B | C But that B is terribly slow, and thus you want to parallelize that. Then you can do: A | parallel --pipe B | C GNU Parallel ...


8

GNU Parallel is specifcally designed to solve this problem: echo -n $IPs | parallel -d ' ' -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string' If your IPs are in a file it is even prettier: cat IPs | parallel -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string' To learn more watch the intro video: ...


8

When you write A | B, both processes already run in parallel. If you see them as using only one core, that's probably because either of CPU affinity settings (perhaps there is some tool to spawn a process with different affinity) or because one process isn't enough to hold a whole core, and the system "prefers" not to spread out computing. To run several ...


7

In addition to solutions already proposed, you can create a makefile that describes how to make a compressed file from uncompressed, and use make -j 4 to run 4 jobs in parallel. The problem is that you will need to name compressed and uncompressed files differently, or store them in different directories, else writing a reasonable make rule will be ...


7

Using GNU Parallel: #!/bin/bash N=$1 n=$2 arr=($( # Generate all combinations of 1..n and 1..N parallel -k --tag /path/to/a.out {1} {2} '|' bc :::: <(seq $N) <(seq $n) | perl -ane 'BEGIN{$min=1e30} $last||=$F[0]; if($F[0] != $last) {print $min,"\n";$min=1e30;$last=$F[0]} $min = $F[2]<$min ? $F[2] : $min; END {print $min,"\n"}' )) echo ${arr[*]} ...


7

Using GNU Parallel, $ parallel -j ${jobs} wget < urls.txt or xargs from GNU Findutils, $ xargs -P ${jobs} wget < urls.txt where ${jobs} is the maximum number of wget you want to allow to run concurrently. Without -j/-P, parallel will run as many jobs at a time as CPU cores (which doesn't necessarily make sense for wget bound by network IO), ...


7

A classic case of RTFM (all of it!). The -T option to GNU tar will read the files to be archived from another file (in my case, /dev/stdin, you can also use -), and there's even a --remove-files option: alias magic_otf_compressor='tar --create -T - --remove-files -O | pixz' (using the parallel version of xz for compression, but you can use your preferred ...


7

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel: ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > ...


6

If you want to parallelize on a machine with multiple cores, you can just use (GNU) xargs, e.g.: echo seq_[0-9][0-9].gz | xargs -n 1 -P 16 ./crunching Meaning: xargs starts up to 16 processes in parallel of ./crunching using 1 token from stdin for each process. You can also use split in combination with xargs. Or you can create a simple Makefile for Job ...


6

(I would make this a comment, but I lack the privileges.) You can also probably just use parallel --gnu. You said that Ubuntu 12(.04?) uses Tollef's version, but it's actually the gnu implementation, it just defaults to Tollef's syntax by default for some reason. I added alias parallel='parallel --gnu' to my login script so I don't have to fuss with it ...


5

To list all files start with number in a directory, find . -maxdepth 1 -regextype "posix-egrep" -regex '.*/[0-9]+.*\.mp3' -type f Problem with your approach is that the find returns a relative path of a file and you are just expecting a filename itself.


5

Simply remove the ; character, so in final : for i in *; do something.py $i & done And for running N instance of your script at the same time, see man 1 parallel See http://www.gnu.org/software/parallel/


5

Why don't you just fork (aka. background) them? foo () { local run=$1 fsl5.0-flirt -in $kar"deformed.nii.gz" -ref normtemp.nii.gz -omat $run".norm1.mat" -bins 256 -cost corratio -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12 fsl5.0-flirt -in $run".poststats.nii.gz" -ref $kar"deformed.nii.gz" -omat $run".norm2.mat" -bins 256 -cost ...


5

Putting multiple jobs in the background is a good way of using the multiple cores of a single machine. parallel however, allows you to spread jobs across multiple servers o your network. From man parallel: GNU parallel is a shell tool for executing jobs in parallel using one or more computers. The typical input is a list of files, a list of ...


5

lftp would do this with the command mirror -R -P 20 localpath - mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once. As explained in man lftp: mirror [OPTS] [source [target]] Mirror specified source directory to local target directory. If target directory ends with a ...


4

I am not aware of any shell that allows you to manually create new threads, usually you can only utilise the existing threads in the current shell (or create subshells, which are indeed new processes). Use python or another language instead. Even if you could, I really would not recommend using a shell script for something on this scale. The performance ...


4

This should do the trick: echo -n $IPs | xargs --max-args=1 -I {} --delimiter ' ' --max-procs=0 \ sh -c "wget -q -O- 'http://{}/somepage.html' | egrep --count '^string'" | \ { NUM=0; while read i; do NUM=$(($NUM + $i)); done; echo $NUM; } The idea here is to make separate counts and sum these at the end. Might fail if the separate counts are big ...


4

GNU Parallel does that and more (using ssh). It can even deal with mixed speed of machines, as it simply has a queue of jobs, that are started on the list of machines (e.g. one per CPU core). When one jobs finishes another one is started. So it does not divide the jobs into clusters before starting, but does it dynamically. Watch the intro videos to learn ...


4

In these cases I'd rather open another terminal. What is the reason that you don't want to do that? Downside of running qsub, is that you have to write a tiny script file for a trivial operation, which takes you some time. I don't know how many other users are working on the same network, but the purpose is meant as a scheduler for jobs of several users on ...


4

Use GNU Parallel: find . -name *.gz | parallel --files 'zcat {} | sort' | parallel -X -j1 sort -m {} ';' rm {} > sorted You can install GNU Parallel simply by: wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel chmod 755 parallel Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 and ...


4

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this: parallel ./pngout -s0 {} R{} ::: *.png You can install GNU Parallel simply by: wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel chmod 755 parallel cp parallel sem Watch the intro videos for GNU Parallel to learn more: ...


4

This is possible and does occur in reality. Use a lock file to avoid this situation. An example, from said page: if mkdir /var/lock/mylock; then echo "Locking succeeded" >&2 else echo "Lock failed - exit" >&2 exit 1 fi # ... program code ... rmdir /var/lock/mylock



Only top voted, non community-wiki answers of a minimum length are eligible