4

I have tens of thousands of gzipped text files, each of which has already been sorted internally.

I've written a script to unzip them (creating a copy) and then using sort -m to merge them.

It all works very well, but in trying to improve the performance I noted that the unzipping operation is about 16-20% of the total run time.

Basically, the merge operation has to scan through the entire bulk of the text files repeatedly, merging 16 files into 1 at a time. Typically, this leads to about 4-5 full scans through the data I have. The unzipping essentially adds one additional scan through it all.

Saving 1 out of 5-6 scans through 50-100 GB of data is a substantial saving.

Thus I started wondering if I could feed the output of gunzip straight into sort (or some other tool) and basically combine the unzipping round with the first round of merging. The output would not be gzipped.

Does anyone know if this can be done? Ideally with the standard tools, but I'm also open to a third party tools if it fits the bill.

Improve this question
5
  • in you question it said that you have thousands of gz'ipped text files that have already been sorted - if they are sorted - why are you sort - merging them again ? .... to be sure just a check if the data is sorted is enough ... then append or concatenate the data to the master file .. ?
    – Nitin4873
    Commented Jun 11, 2013 at 17:00
  • Also , have you tried using the parellel run ? like : sort --parallel=4 -S 10M (Run no more than 4 sorts concurrently, using a buffer size of 10M) ..... saw this in the info doc.
    – Nitin4873
    Commented Jun 11, 2013 at 17:02
  • @NSD To clarify, each file is sorted internally, it is not the case that entire dataset is sorted, with segment 1 being in the first file, etc. The merge process creates a single file that is entirely sorted and can be used to search the data more quickly than doing searches of all the separate files. I've edited my question to clarify this.
    – Kris
    Commented Jun 11, 2013 at 18:56
  • the sort utility uses the merge sort to merge which works well in parallel ..... did you try using parallel processes as i had mentioned before and time them on a small sample size with respect to your previous approach ? ....i forgot to mention also there is a limit of 4 parallel processes when you use sort though.
    – Nitin4873
    Commented Jun 11, 2013 at 21:53
  • did you get a solution for that ? ..... what did you do to reduce the cycle time ? I work with huge files too ..... generally it takes days to get things processed . Hence i was interested in the answer.
    – Nitin4873
    Commented Jun 12, 2013 at 6:18

1 Answer 1

Reset to default
5

You can—the program zcat (or gzip -cd) to write the decompressed data to stdout. From there, you can use bash's process substitution. 1.gz is the numbers 1 through 3; a.gz is the letters a through c; and z.gz is the number 4 and letter d:

$ sort -m <(zcat 1.gz) <(zcat a.gz) <(zcat z.gz)
1
2
3
4
a
b
c
d

Hopefully your data is somewhat more interesting.

Improve this answer
3
  • Even better, you can (g)zcat multiple files: zcat *gz | sort
    – bahamat
    Commented Jun 11, 2013 at 19:55
  • i think thats what @Kris has in his script :(
    – Nitin4873
    Commented Jun 11, 2013 at 21:40
  • 1
    @bahamat no, that won't work. sort -m merges together two or more already sorted files. zcat *.gz would give one "file", which isn't sorted. You'd have to drop -m, and merging already sorted files is (computationally) easier than sorting an unsorted file. Takes much less memory and only a single I/O pass, too. [With a normal sort, you can either blow a lot of memory, or use multiple passes.]
    – derobert
    Commented Jun 11, 2013 at 21:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .