5

In a folder containing X files, I need to concatenate Y files (where X > Y) together into a single text file. I have the filenames (of Y files) that I need to concatenate.

  • In what format do you have the Y files? – forcefsck Feb 16 '12 at 19:54
  • @forcefsck: They are in text format as well – name_masked Feb 16 '12 at 20:03
9

You can use the cat command (see man cat for more information) to concatenate the text files.

If you want to create a new file

cat [FILE1] [FILE2]... > new_file

or if you want to append to an existing file use it like this

cat [FILE1] [FILE2]... >> file 
  • 1
    This, however, does not show how to read the filenames from a list and apply cat to them. – Kusalananda Apr 11 '18 at 17:18
7

If the Y filenames are listed in a list file, a simple combination of xargs and cat is enough:

xargs cat <list >>concatenation_of_files

In the case you've been careful and you've listed files one per line (to avoid problems with spaces in filenames), then just add a -d delimiter option:

xargs -d'\n' cat <list >>concatenation_of_files

(This assumes concatenation_of_files is initially inexistent or empty).

  • Cool! Obvious but didn't see it. – forcefsck Feb 17 '12 at 21:48
  • 1
    You may use >outputfile to truncate the output file. This is ok as it's the output from xargs that is redirected, not from the individual cat invocations. – Kusalananda Apr 11 '18 at 17:15
0

Answers are using bash.

Let's say you want to concat json files in your directory:

cat *json > all.json

Including subdirectories:

shopt -s globstar
cat **/*json > all.json

However, if you have thousands to millions of files, you will encounter:

bash: /bin/cat: Argument list too long

In which case, do the following which also outputs the current file being processed:

shopt -s globstar
rm concat.json
find . -path "**/*.json" | while read -r file; do
    echo -ne "\\r$file"
    cat "$file" >> concat.json
done

Posted on this question, as this other question became closed.

  • This does not read the filenames from a list. – Kusalananda Apr 11 '18 at 17:16
  • Thanks for posting an answer but, yes, this isn't answering the question. Also, you can just do for f in **/*.json; do cat "$f" >> concat.json or find . -path "**/*.json" -exec cat {} >> concat.json . – terdon Apr 11 '18 at 17:31
  • Problem is the question that this answers was marked as a duplicate of this question, which it is not exactly. Seems this question is then intended to become the one for accomplish its aims in the most broadest of sense. – balupton Apr 11 '18 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.