Tag Info

New answers tagged


I don't have the 50 reputation privilege to comment on the above answer https://unix.stackexchange.com/a/104008/125690, so just as a comment to Ole Tange's response: If you're using a non-bash shell you'll need to also export SHELL=/bin/bash before running parallel otherwise you'll get an error like: fish: Unknown command 'myfunc arg' fish: myfunc arg


With GNU Parallel it looks like this: parallel tar jcvf /tmp/{= s:/$:: =}.tar.bz2 {} ::: */ or: parallel tar jcvf /tmp/{}.tar.bz2 {} ::: * For better compression try: parallel tar -I pxz -cvf /tmp/{= s:/$:: =}.tar.xz {} ::: */


pbzip2 works quite well. As with the answer above, tar to stdout and pipe to pbzip2: $ tar -cf - mydir/ | pbzip2 > mydir.tar.bz2 pbzip2 accepts multiple options that allow for adjusting number of processors, amount of memory used, level of compression etc. http://compression.ca/pbzip2/ Or for one archive per directory (assumes no spaces or special ...


Just tar to stdout and pipe it to pigz. (You most likely don't want to parallelize disk access, just the compression part.): $ tar c- myDirectory/ | pigz > myDirectory.tar.gz A plain tar invocation like the one above basically only concatenates directory trees in a reversible way. The compression part can be separate as it is in this example. pigz ...


Sample task task(){ sleep 0.5; echo "$1"; } Sequential runs for thing in a b c d e f g; do task "$thing" done Parallel runs for thing in a b c d e f g; do task "$thing" & done Parallel runs in N-process batches N=4 ( for thing in a b c d e f g; do ((i=i%N)); ((i++==0)) && wait task "$thing" & done )


for stuff in things do sem -j+0 ( something with stuff ) done sem --wait This will use semaphores, parallelizing as many iterations as the number of available cores (-j +0 means you will parallelize N+0 jobs, where N is the number of available cores). sem --wait tells to wait until all the iterations in the for loop have terminated execution before ...

Top 50 recent answers are included