7

In a folder containing X files, I need to concatenate Y files (where X > Y) together into a single text file. I have the filenames (of Y files) that I need to concatenate.

2
  • In what format do you have the Y files?
    – forcefsck
    Feb 16, 2012 at 19:54
  • @forcefsck: They are in text format as well Feb 16, 2012 at 20:03

4 Answers 4

9

You can use the cat command (see man cat for more information) to concatenate the text files.

If you want to create a new file

cat [FILE1] [FILE2]... > new_file

or if you want to append to an existing file use it like this

cat [FILE1] [FILE2]... >> file 
1
  • 2
    This, however, does not show how to read the filenames from a list and apply cat to them.
    – Kusalananda
    Apr 11, 2018 at 17:18
8

If the Y filenames are listed in a list file, a simple combination of xargs and cat is enough:

xargs cat <list >>concatenation_of_files

In the case you've been careful and you've listed files one per line (to avoid problems with spaces in filenames), then just add a -d delimiter option:

xargs -d'\n' cat <list >>concatenation_of_files

(This assumes concatenation_of_files is initially inexistent or empty).

2
  • Cool! Obvious but didn't see it.
    – forcefsck
    Feb 17, 2012 at 21:48
  • 1
    You may use >outputfile to truncate the output file. This is ok as it's the output from xargs that is redirected, not from the individual cat invocations.
    – Kusalananda
    Apr 11, 2018 at 17:15
4

Answers are using bash.

Let's say you want to concat json files in your directory:

cat *json > all.json

Including subdirectories:

shopt -s globstar
cat **/*json > all.json

However, if you have thousands to millions of files, you will encounter:

bash: /bin/cat: Argument list too long

In which case, do the following which also outputs the current file being processed:

shopt -s globstar
rm concat.json
find . -path "**/*.json" | while read -r file; do
    echo -ne "\\r$file"
    cat "$file" >> concat.json
done

Posted on this question, as this other question became closed.

3
  • This does not read the filenames from a list.
    – Kusalananda
    Apr 11, 2018 at 17:16
  • Thanks for posting an answer but, yes, this isn't answering the question. Also, you can just do for f in **/*.json; do cat "$f" >> concat.json or find . -path "**/*.json" -exec cat {} >> concat.json .
    – terdon
    Apr 11, 2018 at 17:31
  • Problem is the question that this answers was marked as a duplicate of this question, which it is not exactly. Seems this question is then intended to become the one for accomplish its aims in the most broadest of sense.
    – balupton
    Apr 11, 2018 at 19:22
0

(I know this is REALLY late and not actually an answer but a suggestion if you are doing this.)

Do not use the same file extension as the files that you are concatenating.

cat *.json > all.json.txt

Then come back and...

mv all.json.txt all.json

Not actually sure if this solves the problem in the question but this will avoid the looping issue.

1
  • Welcome to the site, and thank you for your contribution. Unfortunately, it is not clear which "looping issue" is being avoided; please consider adding more explanation.
    – AdminBee
    Dec 1, 2021 at 9:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .