Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have tens of thousands of gzipped text files, each of which has already been sorted internally.

I've written a script to unzip them (creating a copy) and then using sort -m to merge them.

It all works very well, but in trying to improve the performance I noted that the unzipping operation is about 16-20% of the total run time.

Basically, the merge operation has to scan through the entire bulk of the text files repeatedly, merging 16 files into 1 at a time. Typically, this leads to about 4-5 full scans through the data I have. The unzipping essentially adds one additional scan through it all.

Saving 1 out of 5-6 scans through 50-100 GB of data is a substantial saving.

Thus I started wondering if I could feed the output of gunzip straight into sort (or some other tool) and basically combine the unzipping round with the first round of merging. The output would not be gzipped.

Does anyone know if this can be done? Ideally with the standard tools, but I'm also open to a third party tools if it fits the bill.

share|improve this question
    
in you question it said that you have thousands of gz'ipped text files that have already been sorted - if they are sorted - why are you sort - merging them again ? .... to be sure just a check if the data is sorted is enough ... then append or concatenate the data to the master file .. ? –  nsd Jun 11 '13 at 17:00
    
Also , have you tried using the parellel run ? like : sort --parallel=4 -S 10M (Run no more than 4 sorts concurrently, using a buffer size of 10M) ..... saw this in the info doc. –  nsd Jun 11 '13 at 17:02
    
@NSD To clarify, each file is sorted internally, it is not the case that entire dataset is sorted, with segment 1 being in the first file, etc. The merge process creates a single file that is entirely sorted and can be used to search the data more quickly than doing searches of all the separate files. I've edited my question to clarify this. –  Kris Jun 11 '13 at 18:56
    
the sort utility uses the merge sort to merge which works well in parallel ..... did you try using parallel processes as i had mentioned before and time them on a small sample size with respect to your previous approach ? ....i forgot to mention also there is a limit of 4 parallel processes when you use sort though. –  nsd Jun 11 '13 at 21:53
    
did you get a solution for that ? ..... what did you do to reduce the cycle time ? I work with huge files too ..... generally it takes days to get things processed . Hence i was interested in the answer. –  nsd Jun 12 '13 at 6:18

1 Answer 1

up vote 5 down vote accepted

You can—the program zcat (or gzip -cd) to write the decompressed data to stdout. From there, you can use bash's process substitution. 1.gz is the numbers 1 through 3; a.gz is the letters a through c; and z.gz is the number 4 and letter d:

$ sort -m <(zcat 1.gz) <(zcat a.gz) <(zcat z.gz)
1
2
3
4
a
b
c
d

Hopefully your data is somewhat more interesting.

share|improve this answer
    
Even better, you can (g)zcat multiple files: zcat *gz | sort –  bahamat Jun 11 '13 at 19:55
    
i think thats what @Kris has in his script :( –  nsd Jun 11 '13 at 21:40
1  
@bahamat no, that won't work. sort -m merges together two or more already sorted files. zcat *.gz would give one "file", which isn't sorted. You'd have to drop -m, and merging already sorted files is (computationally) easier than sorting an unsorted file. Takes much less memory and only a single I/O pass, too. [With a normal sort, you can either blow a lot of memory, or use multiple passes.] –  derobert Jun 11 '13 at 21:50
    
That works a treat! Thanks @derobert –  Kris Jun 12 '13 at 8:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.