This question already has an answer here:

Is there a utility that dumps a file line by line to stdout and deletes those lines?

I have a number of large files that I want to merge via cat. Of course I could do

cat *.txt > merged.txt 

But this will take double the space originally occupied by the txt files. Deleting them after cat like so

cat *.txt > merged.txt && rm -rf *.txt

reduces the storage requirements but still needs 2x the space to be present in the meantime

I'm looking for a command that would stream lines to stdout while at the same time deleting them in the original files:

<some command> *.txt > merged.txt

such that the total space used by *.txt + merged.txt and is never more that what *.txt needed originally

marked as duplicate by Satō Katsura, thrig, HalosGhost, techraf, countermode Oct 19 '16 at 14:52

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.


Line by line sounds a bit tricky, but if you accept to do it file by file (so that storage never exceeds all *.txt files + one of them:

find -name "*.txt" -exec cat {} >> result.txt \; -exec \rm -f {} \;

BE VERY CAREFUL! Always check the result of your find command before running it, or you'll be rm'ing stuff that should not be deleted at lightspeed.

Not the answer you're looking for? Browse other questions tagged or ask your own question.