Tag Info

New answers tagged

0

I don't have the 50 reputation privilege to comment on the above answer https://unix.stackexchange.com/a/104008/125690, so just as a comment to Ole Tange's response: If you're using a non-bash shell you'll need to also export SHELL=/bin/bash before running parallel otherwise you'll get an error like: fish: Unknown command 'myfunc arg' fish: myfunc arg


0

With GNU Parallel it looks like this: parallel tar jcvf /tmp/{= s:/$:: =}.tar.bz2 {} ::: */ or: parallel tar jcvf /tmp/{}.tar.bz2 {} ::: * For better compression try: parallel tar -I pxz -cvf /tmp/{= s:/$:: =}.tar.xz {} ::: */


1

pbzip2 works quite well. As with the answer above, tar to stdout and pipe to pbzip2: $ tar -cf - mydir/ | pbzip2 > mydir.tar.bz2 pbzip2 accepts multiple options that allow for adjusting number of processors, amount of memory used, level of compression etc. http://compression.ca/pbzip2/ Or for one archive per directory (assumes no spaces or special ...


3

Just tar to stdout and pipe it to pigz. (You most likely don't want to parallelize disk access, just the compression part.): $ tar c- myDirectory/ | pigz > myDirectory.tar.gz A plain tar invocation like the one above basically only concatenates directory trees in a reversible way. The compression part can be separate as it is in this example. pigz ...


0

Sample task task(){ sleep 0.5; echo "$1"; } Sequential runs for thing in a b c d e f g; do task "$thing" done Parallel runs for thing in a b c d e f g; do task "$thing" & done Parallel runs in N-process batches N=4 ( for thing in a b c d e f g; do ((i=i%N)); ((i++==0)) && wait task "$thing" & done )


2

for stuff in things do sem -j+0 ( something with stuff ) done sem --wait This will use semaphores, parallelizing as many iterations as the number of available cores (-j +0 means you will parallelize N+0 jobs, where N is the number of available cores). sem --wait tells to wait until all the iterations in the for loop have terminated execution before ...



Top 50 recent answers are included