Create new concatenated files of same name in multiple directories

Question

I have many files of the same name distributed in many sub-directories with different names (though all at the same level). I'd like to concatenate all the files of the same into a new file with that name. I'd like this new file to be in the parent directory.

I've already tried a few answers posted here on SE: How to move files with same name and concatenate

Here's what I've tried and the issues:

find */*/*/seq/in/ -type f -name '*.fasta' -exec bash -c 'cat "{}" >> new_file' \;

This concatenated all files with suffix .fasta, into one file call new_file, not just those with matching names.

for file in */*/*/seq/in/*.fasta; 
do 
cat "$file" >> "$file.cat" done

This just made a copy of each file in the same original sub-directory and appended it with .cat

What am I doing wrong? Thank you so much!

Answer 1

You will have to do this in two steps:

1. Find all unique Fasta filenames.
2. For each name, find all the files with that name and concatenate them.

With code:

Find all Fasta files (this assumes sane filename with no embedded newlines):

find . -type f -path '*/seq/in/*.fasta' -exec basename {} ';' | sort -u -o file.list

Then for each name, concatenate all files with the same name. The resulting files will be placed in the new directory in the current directory. The name of the new file will be the same as that of the concatenated files.

mkdir new
while read -r name; do
    find . -type f -path "*/seq/in/$name" -exec cat {} + >"new/$name"
done <file.list

The thing that you might want to add to each of the find invocations is -mindepth 6 -maxdepth 6 (or the appropriate depth, 6 may be off by one), so that the files are all taken from the same depth in the file hierarchy. Insert these options just before -type f.

---

Alternative suggested by steeldriver in comments, which bypasses the creation of a separate list of Fasta filenames:

mkdir new
find . -type f -path '*/seq/in/*.fasta' \
    -exec sh -c 'for pathname do cat "$pathname" >>"new/${pathname##*/}"; done' find-sh {} +