Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have 1000 gzipped files which I want to sort.

Doing this sequentially, the procedure looks pretty straightforward:

find . -name *.gz -exec zcat {} | sort > {}.txt \;

Not sure that the code above works (please correct me if I did a mistake somewhere), but I hope you understand the idea.

Anyway, I'd like to parallelize ungzip/sort jobs in order to make the whole thing faster. Also, I don't want to see all 1000 processes running simultaneously. It would be great to have some bounded job queue (like BlockingQueue in Java or BlockingCollection in .NET) with configurable capacity. In this case, only, say, 10 processes will run in parallel.

Is it possible to do this in shell?

share|improve this question
    
Is there some critical reason you need to do this in shell? A language like Python is not very different in terms of convenience, and has good facilities for this sort of thing. –  Faheem Mitha Dec 6 '11 at 7:58
2  
use xargs with multiprocessing jobs option. –  Nikhil Mulley Dec 6 '11 at 8:00
    
+Faheem Mitha, there is no critical reason. I just wondering if it is possible in shell. –  DNNX Dec 6 '11 at 8:07

3 Answers 3

up vote 0 down vote accepted

A quick trip to Google reveals this interesting approach: http://pebblesinthesand.wordpress.com/2008/05/22/a-srcipt-for-running-processes-in-parallel-in-bash/

share|improve this answer
    
Please don't just post a link as an answer, both because the linked site may be unavailable, and because as it is your answer doesn't show what it's about. Summarize the essential parts of the answer here, and provide the link for reference. –  Gilles Dec 6 '11 at 23:51

Use GNU Parallel:

find . -name *.gz | parallel --files 'zcat {} | sort' | parallel -X -j1 sort -m {} ';' rm {} > sorted

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 and walk through the tutorial (man parallel_tutorial). You command line with love you for it.

share|improve this answer

I'd pick make(1) for this task -- it isn't shell, but the make(1) jobserver is nearly exactly what you wanted, and this task is well-suited to make(1)'s abilities. Note that the line starting gzip -cd is indented with a tab character. This is vital. (make(1) can also feel a little old at times.)

$ cat Makefile 
TXT := $(wildcard *.gz)

all: $(TXT:.gz=.txt)

%.txt:%.gz
    gzip -cd $< | sort > $@
$ cp /usr/share/man/man2/*.gz .
$ ls -l
total 1992
-rw-r--r-- 1 sarnold sarnold  4447 2011-12-06 00:22 aa_change_hat.2.gz
-rw-r--r-- 1 sarnold sarnold  3977 2011-12-06 00:22 aa_change_profile.2.gz
-rw-r--r-- 1 sarnold sarnold  5082 2011-12-06 00:22 accept.2.gz
...
$ time make -j 10
gzip -cd aa_change_hat.2.gz | sort > aa_change_hat.2.txt
gzip -cd aa_change_profile.2.gz | sort > aa_change_profile.2.txt
gzip -cd accept.2.gz | sort > accept.2.txt
gzip -cd accept4.2.gz | sort > accept4.2.txt
gzip -cd access.2.gz | sort > access.2.txt
...
gzip -cd write.2.gz | sort > write.2.txt
gzip -cd writev.2.gz | sort > writev.2.txt

real    0m0.259s
user    0m0.190s
sys 0m0.020s
$ rm w*txt
$ make
gzip -cd wait.2.gz | sort > wait.2.txt
gzip -cd wait3.2.gz | sort > wait3.2.txt
gzip -cd wait4.2.gz | sort > wait4.2.txt
gzip -cd waitid.2.gz | sort > waitid.2.txt
gzip -cd waitpid.2.gz | sort > waitpid.2.txt
gzip -cd write.2.gz | sort > write.2.txt
gzip -cd writev.2.gz | sort > writev.2.txt
$ 

Notice with the rm w*txt command that make(1) intelligently only does the minimum amount of work necessary to get anything done.

share|improve this answer
    
I have up to 100 experiments to run in Matlab. But available CPU cores are limited. This answer is exactly what I need. –  onemach Jan 25 at 6:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.