concatenate all the text files in all the sub folders into one giant text file

Question

Ok so the situation is that I have an unknown number of sub-directories that all follow the same naming profile

folder0, folder1, folder2, folder3 etc

Now each folder will have 3 text files and these text files will have the same 3 file names in all folders

file1 file2 file3

I would like to find an easy way to concatenate all the text files in all the folders into one text file, starting in folder0 with file1, then file2, then file3 and the same order in all folders.

Now for a small number of folders I could use cat

cat folder0/file1 folder0/file2 folder0/file3 folder1/file1 folder1/file2 folder1/file3 folder2/file1 folder2/file2 folder2/file3 folder3/file1 folder3/file2 folder3/file3 > textfile

but the number of folders is unknown and could range into the 100's or 1000's

anybody have any indea of a script that would acomplish this.

Answer 1

With only unix tools and pipes:

find ./ -type f -regex './folder[0-9]+/file[0-9]+' -print0 | sort -zV | xargs --null cat

Explanation:

find searches for all files matching some condition

1. ./ tells find to search from where you are currently at, i.e. your working directory
2. -type f tells find to only find files, in case there ever is a directory named as a file
3. -regex tells find to only search for files where the whole path of the file matches the given pattern. In this case "./folder" followed by one or more digits, followed by a new directory and then "file" followed by one or more digits.
4. -print0 tells find to output the files it found separated by a null character, which is guaranteed to not appear in file names and will work even if your file names contain newlines, which they could

sort sorts the list it gets

1. the -z tells sort to sort null character separated entries instead of newline separated ones, as would be the default
2. the -V tells sort to sort the entries as a human would intuitively sort them, i.e. file11 comes after both file1 and file2, not in the middle.

xargs just uses the entries it gets and passes them as argument to cat. --null tells xargs, you guessed it, to tell apart different entries by a null character between them, not blanks or newlines.

Answer 2

With zsh:

autoload zargs
zargs -r -- folder<->/file<1-3>(nN) -- cat > hugefile

With bash, and with GNU tools:

shopt -s extglob nullglob
print0() { (( $# == 0 )) || printf '%s\0' "$@"; }
 
xargs -r0a <(
  print0 folder+([0123456789])/file[123] | sort -zV
  ) cat > hugefile

Where +([0123456789]) (with extglob which enables some of ksh's advanced glob operators, including +(...)) is the equivalent of zsh's <-> (any sequence of ASCII decimal digits), GNU sort -V does the equivalent of zsh's n glob qualifier (for numeric sorting so that folder10 comes after folder9 and not between folder1 and folder2 with the default alphabetical sorting), nullglob for the N glob qualifier and GNU xargs is used in place of zsh's zargs to split the list to avoid arg list too long errors.

We define a print0 function that prints its arguments NUL-delimited (as bash doesn't have zsh's print -rNC1), though here, given that file paths won't contain any of the characters that are special to xargs without -0, we might as well print them newline separated with printf '%s\n' and ignore that fact that it would print an empty line if the list was empty given that xargs will ignore that line anyway.

Answer 3

Up to a certain number of folders, you can do:

cat folder*/* > file

Or, to target specific files/directories only:

cat {folder1,folder2,folder3}/{file1,file2,file3} > file

If the names really do consist of a fixed string and sequential numbers, you can further simplify to

cat folder{1..3}/file{1..3} > file

Now, this will fail if you have a huge number of files and folders, but it should work for most cases. For instance, on my machine, I created 3000 files in 1000 directories with:

mkdir folder{1..1000}
touch folder{1..1000}/file{1..3}
for f in */*; do echo "$f" > $f; done

And then concatenated all files into one with:

cat folder*/* > file

Which gives a file with 3000 lines.

If, however, you run into issues with this, you can always use find:

find folder1 folder2 folder3 -name "file[123]" -exec cat {} + > file

Or

find folder* -name "file[123]" -exec cat {} + > file

CAVEAT: none of these approaches will let you specify the order, but they should have the same order in all directories.