8

I have a large folder containing many sub-directories each holding many .txt files. I want to concatenate all of these files into one .txt file. I am able to do it for each of the sub-directories with cat *.txt>merged.txt, but I am trying to do it for all of the files in the large folder. How do I do this?

3 Answers 3

13

try with

find /path/to/source -type f -name '*.txt' -exec cat {} + >mergedfile

find all '*.txt' files in /path/to/source recursively for sub-directories and concatenate all into one mergedfile.

To concatenate each sub-directories files within its directory, do:

find . -mindepth 1 -type d -execdir sh -c 'cat $1/*.txt >> $1/mergedfile' _ {} \;
14
  • >> can be > in the first find call.
    – Kusalananda
    Jun 4, 2018 at 5:36
  • @Kusalananda won't that truncate the mergedfile if ARG_MAX exceed? Jun 4, 2018 at 5:49
  • The > redirects the output of find, not cat. The cat command ends at the +, and you can't do redirections in -exec without using a child shell (sh -c). In your second example, you won't need it either as you do one directory at a time.
    – Kusalananda
    Jun 4, 2018 at 5:52
  • Actually, that second example won't work. Since -execdir is already executing with the directory as the working directory, you should get rid of $1/ in the command.
    – Kusalananda
    Jun 4, 2018 at 5:56
  • 1
    @DerekMahar _ there is 0th argument to the sh -c '....' and {} is the 1st. when you remove the _, the {} is being 0th argument while the sh -c '...' perform and do stuff on the $1 argument but now there is no 1st ($1) argument. why we don't use the {} as the first argument because in general always the 1st argument is the script name and all errors/warning/... will use that name prefixed to alert where things go wrong. Mar 29, 2023 at 14:30
2

If you are using Bash and the number of text files is contained (i.e. does not exceed the maximum argument number limit, which is very large but not infinite), you can easily achieve this with the globstar feature:

$ shopt -s globstar
$ cat **/*.txt > merged.txt

A more general, although less elegant approach, will be to use find as the driver and make it call cat on each file, appending the output:

$ find -name \*.txt -exec sh -c 'cat {} >> merged.out' \;

Calling sh is needed here because you want to append the result of each cat. Make sure the output file has a different extension or lies outside of the tree you're merging, or find may try to concatenate the output with itself.

1

If you have to do the concatenation in a particular order, then the below will concatenate the files in lexicographical order (sorted by pathnames) in bash:

shopt -s globstar
for name in **/*.txt; do
    [ -f "$name" ] && cat <"$name"
done >merged.out

This is similar to the find command

find  . -type f -name '*.txt' -exec cat {} ';' >merged.out

except that the ordering may be different, symbolic links to regular files would be included (add a && [ ! -L "$name" ] if you don't want them) and hidden files (and files in hidden directories) would be excluded (use shopt -s dotglob to add them back).

4
  • what else your first command does that the same in my answer doesn't? Jun 4, 2018 at 6:14
  • @αғsнιη Absolutely nothing now when you've changed your answer. I will modify that part. Thanks for letting me know.
    – Kusalananda
    Jun 4, 2018 at 6:26
  • Does bash guarantee that **/*.txt sorts the pathnames in lexicographical order? Mar 29, 2023 at 22:27
  • 1
    @DerekMahar Yes, the list resulting from expanding a globbing pattern is guaranteed to be lexicographically sorted. From the POSIX standard: "If the pattern matches any existing filenames or pathnames, the pattern shall be replaced with those filenames and pathnames, sorted according to the collating sequence in effect in the current locale."
    – Kusalananda
    Mar 29, 2023 at 22:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .