Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to merge bunch of files in a directory on a unix box that has 8 cores. I am looking for a fast way to merge these files. The cat command uses only one core... is there a way where I can make it use multiple cores for parallel processing.

share|improve this question

3 Answers 3

There is no effective way to parallelize an append I/O operation; each line must be written in turn.

share|improve this answer
1  
Well, if you know the file sizes, I suppose you could in theory create a big sparse file and then chunk the parts into the right offsets. However, this is still going to be I/O bound, so it doesn't offer any benefits. –  tripleee Jul 4 '13 at 5:30

First of all, confirm the performance bottle-neck. Multi-thread is useless if disk io is your bottle-neck.

If you have a really fast disk or SSD, you can stat the files' lengths, calculate offsets, and use several dd commands.

share|improve this answer

How about a script like this (with the sleeps to demonstrate how it parallelizes operations):

#!/bin/bash

eval exec 3\<<(echo FIRST FILE; sleep 3)
eval exec 4\<<(echo SECOND FILE; sleep 3)

cat <&3
cat <&4

It parallelizes the reading of the input files. Note this is limited in many ways, and won't parallelize the output, but that can't be done anyway.

Here's a more complete example which processes all files in the current directory:

#!/bin/bash

fd=3
for file in *
do
    eval exec $fd\<<(cat $file)

    fd=$((fd + 1))
done

out_fd=3
while [ "$out_fd" -lt "$fd" ]
do
    cat <&${out_fd}
    out_fd=$((out_fd + 1))
done

The points above about performance are good; there's a chance this won't help at all.

Update On further thought, this will only prefetch up to the pipe buffer size, 64k. The following modification will pull all of the files into memory; if they can't fit, don't use it:

#!/bin/bash

fd=3
for file in *
do
    eval exec $fd\<<(content=$(<$file); echo "$content")

    fd=$((fd + 1))
done

out_fd=3
while [ "$out_fd" -lt "$fd" ]
do
    cat <&${out_fd}
    out_fd=$((out_fd + 1))
done
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.