Concatenating thousands of files: > vs >>

Question

I found two seemingly contradictory answers on StackOverflow to the following questions:

• Concatenating Thousands of Text Files Across Hundreds of Directories (while keeping some structure)
• How do I concatenate files in a subdirectory with Unix find execute and cat into a single file?

The top answer to the first question suggests:

find . -name *.txt -print0 | xargs -0 cat >> out.txt

while the top answer to the second question suggests:

find . -name *.txt -print0 | xargs -0 cat > out.txt

As far as I know, the first one is correct since it uses the >> (append) operator, but not the second one since it uses the > operator which I thought simply redirects output to a file. However, the second answer has more votes (10) and was also accepted with no comments. Are both answers correct? Why? What is the purpose of having these two operators then?

Answer 1

The second example:

find . -name '*.txt' -print0 | xargs -0 cat > out.txt

Is completely legal and will recreate the file, out.txt each time it's run, while the first will concatenate to out.txt if it runs. But both commands are doing essentially the same thing.

What's confusing the issue is the xargs -0 cat. People think that the redirect to out.txt is part of that command when it isn't. The redirect is happening after xargs -o cat has taken input in via STDIN, and then cat'ing that output as a single stream out to STDOUT. The xargs is optimizing the cat'ing of the files not their output.

Here's an example that kind of shows what I'm saying. If we insert a pv -l in between the xargs -0 cat and the output to the file out.txt we can see how many lines cat has written.

Example

To show this I created a directory with 10,000 files in it.

for i in `seq -w 1 10000`;do echo "contents of file$i.txt" > file$i.txt;done

Each file looks similar to this:

$ more file00001.txt 
contents of file00001.txt

The output from pv:

$ find . -name '*.txt' -print0 | xargs -0 cat | pv -l > singlefile.rpt
  10k 0:00:00 [31.1k/s] [  <=> 

As we can see, 10k lines were written out to my singlefile.rpt file. If xargs were passing us chunks of output, then we'd see that by a reduction in the number of lines that were being presented to pv.

Answer 2

What is the purpose of having these two operators then?

That's an easy one: Because there are different use cases. Sometimes it is useful to truncate the target file to size 0 first, sometimes (e.g. log files) it makes more sense to append data to a file.

In this case it makes no sense to append. You want a file with exactly the content of the files you selected and not "a file with whatever data at the beginning and the content of the selected files at the end".

Answer 3

I would go with the second one. The redirect of stdout gets caught by bash when you hit enter, so it's not like you create a new redirect for every line of find/xargs (which might have been their thinking). If out.txt doesn't exist they should be identical, if it already has data, then the second one at least resets the file to known content (that is, no content).