5

I have 1000 gzipped files which I want to sort.

Doing this sequentially, the procedure looks pretty straightforward:

find . -name *.gz -exec zcat {} | sort > {}.txt \;

Not sure that the code above works (please correct me if I did a mistake somewhere), but I hope you understand the idea.

Anyway, I'd like to parallelize ungzip/sort jobs in order to make the whole thing faster. Also, I don't want to see all 1000 processes running simultaneously. It would be great to have some bounded job queue (like BlockingQueue in Java or BlockingCollection in .NET) with configurable capacity. In this case, only, say, 10 processes will run in parallel.

Is it possible to do this in shell?

  • Is there some critical reason you need to do this in shell? A language like Python is not very different in terms of convenience, and has good facilities for this sort of thing. – Faheem Mitha Dec 6 '11 at 7:58
  • 2
    use xargs with multiprocessing jobs option. – Nikhil Mulley Dec 6 '11 at 8:00
  • +Faheem Mitha, there is no critical reason. I just wondering if it is possible in shell. – DNNX Dec 6 '11 at 8:07
1

A quick trip to Google reveals this interesting approach: http://pebblesinthesand.wordpress.com/2008/05/22/a-srcipt-for-running-processes-in-parallel-in-bash/

for ARG in  $*; do
    command $ARG &
    NPROC=$(($NPROC+1))
    if [ "$NPROC" -ge 4 ]; then
        wait
        NPROC=0
    fi
done
  • Please don't just post a link as an answer, both because the linked site may be unavailable, and because as it is your answer doesn't show what it's about. Summarize the essential parts of the answer here, and provide the link for reference. – Gilles Dec 6 '11 at 23:51
4

Use GNU Parallel:

find . -name *.gz | parallel --files 'zcat {} | sort' | parallel -X -j1 sort -m {} ';' rm {} > sorted

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 and walk through the tutorial (man parallel_tutorial). You command line with love you for it.

2

I'd pick make(1) for this task -- it isn't shell, but the make(1) jobserver is nearly exactly what you wanted, and this task is well-suited to make(1)'s abilities. Note that the line starting gzip -cd is indented with a tab character. This is vital. (make(1) can also feel a little old at times.)

$ cat Makefile 
TXT := $(wildcard *.gz)

all: $(TXT:.gz=.txt)

%.txt:%.gz
    gzip -cd $< | sort > $@
$ cp /usr/share/man/man2/*.gz .
$ ls -l
total 1992
-rw-r--r-- 1 sarnold sarnold  4447 2011-12-06 00:22 aa_change_hat.2.gz
-rw-r--r-- 1 sarnold sarnold  3977 2011-12-06 00:22 aa_change_profile.2.gz
-rw-r--r-- 1 sarnold sarnold  5082 2011-12-06 00:22 accept.2.gz
...
$ time make -j 10
gzip -cd aa_change_hat.2.gz | sort > aa_change_hat.2.txt
gzip -cd aa_change_profile.2.gz | sort > aa_change_profile.2.txt
gzip -cd accept.2.gz | sort > accept.2.txt
gzip -cd accept4.2.gz | sort > accept4.2.txt
gzip -cd access.2.gz | sort > access.2.txt
...
gzip -cd write.2.gz | sort > write.2.txt
gzip -cd writev.2.gz | sort > writev.2.txt

real    0m0.259s
user    0m0.190s
sys 0m0.020s
$ rm w*txt
$ make
gzip -cd wait.2.gz | sort > wait.2.txt
gzip -cd wait3.2.gz | sort > wait3.2.txt
gzip -cd wait4.2.gz | sort > wait4.2.txt
gzip -cd waitid.2.gz | sort > waitid.2.txt
gzip -cd waitpid.2.gz | sort > waitpid.2.txt
gzip -cd write.2.gz | sort > write.2.txt
gzip -cd writev.2.gz | sort > writev.2.txt
$ 

Notice with the rm w*txt command that make(1) intelligently only does the minimum amount of work necessary to get anything done.

  • I have up to 100 experiments to run in Matlab. But available CPU cores are limited. This answer is exactly what I need. – onemach Jan 25 '14 at 6:37
0

Sorting the uncompressed contents of many compressed files and storing the result in an uncompressed file:

find . -type f -name '*.gz'
    -exec sh -c 'for n; do zcat "$n" | sort -o "$n.txt"; done' sh {} +

This will execute the for loop

for n; do
    zcat "$n" | sort -o "$n.txt"
done

with as many files as possible at once. A for loop that does not have in X will iterate over "$@" by default.

The sh -c shell will be called by find with as many file paths as possible (due to the + instead of \; at the end), and these paths will be available to the sh -c shell in $@.


In your original command,

find . -name *.gz -exec zcat {} | sort > {}.txt \;

you have some issues:

  1. *.gz is unquoted, which means that the shell will perform filename globbing on it with the filenames in the current directory.

  2. -exec can only grok a simple command, not a pipeline.

  3. You don't restrict to regular files, which theoretically means you could pick up a directory whose name is something.gz.

0

With GNU xargs, you can do:

xargs -P4 -n 10 -r0a <(find . -name '*.gz' -type f -print0) sh -c '
  for file do
    zcat < "$file" | sort > "$file.txt"
  done' sh {} +

That would call up to 4 sh in parallel, each processing up to 10 files one after the other in a loop.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.