0

I have a large folder containing many sub-directories each holding many .txt files. I want to concatenate all of these files into one .txt file. I am able to do it for each of the sub-directories with cat *.txt>merged.txt, but I am trying to do it for all of the files in the large folder. How do I do this?

1

try with

find /path/to/source -type f -name '*.txt' -exec cat {} + >mergedfile

find all '*.txt' files in /path/to/source recursively for sub-directories and concatenate all into one mergedfile.

To concatenate each sub-directories files within its directory, do:

find . -mindepth 1 -type d -execdir sh -c 'cat $1/*.txt >> $1/mergedfile' _ {} \;
  • >> can be > in the first find call. – Kusalananda Jun 4 '18 at 5:36
  • @Kusalananda won't that truncate the mergedfile if ARG_MAX exceed? – αғsнιη Jun 4 '18 at 5:49
  • The > redirects the output of find, not cat. The cat command ends at the +, and you can't do redirections in -exec without using a child shell (sh -c). In your second example, you won't need it either as you do one directory at a time. – Kusalananda Jun 4 '18 at 5:52
  • Actually, that second example won't work. Since -execdir is already executing with the directory as the working directory, you should get rid of $1/ in the command. – Kusalananda Jun 4 '18 at 5:56
  • @Kusalananda your first point about using > instead of >> in first command is right but $1/ is needed in second command and that works I tested before. note that execdir is changing for the find not for the child-shell I used there – αғsнιη Jun 4 '18 at 6:10
0

If you are using Bash and the number of text files is contained (i.e. does not exceed the maximum argument number limit, which is very large but not infinite), you can easily achieve this with the globstar feature:

$ shopt -s globstar
$ cat **/*.txt > merged.txt

A more general, although less elegant approach, will be to use find as the driver and make it call cat on each file, appending the output:

$ find -name \*.txt -exec sh -c 'cat {} >> merged.out' \;

Calling sh is needed here because you want to append the result of each cat. Make sure the output file has a different extension or lies outside of the tree you're merging, or find may try to concatenate the output with itself.

0

If you have to do the concatenation in a particular order, then the below will concatenate the files in lexicographical order (sorted by pathnames) in bash:

shopt -s globstar
for name in **/*.txt; do
    [ -f "$name" ] && cat <"$name"
done >merged.out

This is similar to the find command

find  . -type f -name '*.txt' -exec cat {} ';' >merged.out

except that the ordering may be different, symbolic links to regular files would be included (add a && [ ! -L "$name" ] if you don't want them) and hidden files (and files in hidden directories) would be excluded (use shopt -s dotglob to add them back).

  • what else your first command does that the same in my answer doesn't? – αғsнιη Jun 4 '18 at 6:14
  • @αғsнιη Absolutely nothing now when you've changed your answer. I will modify that part. Thanks for letting me know. – Kusalananda Jun 4 '18 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.