Cat similarly named files from different directories into single file with same name

Question

Can anyone help me with the following issue? I have circa 40 directories of different species, each with 100s of sequence files that contain orthologous sequences. The sequence files are similarly named for each of the species directories. I want to concatenate the identically named files of the 40 species directories into a single sequence file which is named similarly.

E.g. I have the following 3 directories: "Species1", "Species2", "Species3". Within these directories are similarly named files: "SequenceA.fasta", "SequenceB.fasta", "SequenceC.fasta". I need to concatenate all contents of the different SequenceA.fasta files into one new file named "SequenceA.fasta" in another directory. How do I do this?

I tried it with the following loop, which failed. Files are created but are empty:

ls . | while read FILE; do cat ./*/"$FILE" >> ./final/"$FILE"; done

Thanks for any advice or help!

(Sorry for any potential cross-posting, I accidentally posted this question in a wrong forum earlier)

Answer 1

There are several things to note in this answer.

1. It is generally a bad idea to parse the output of ls if you can instead use shell glob patterns to do what you want instead - see [http://mywiki.wooledge.org/ParsingLs]. I didn't use nullglob shell option for portability, but that would make the scripts slightly shorter.

2. You want to make sure that your glob patterns are not too broad so you tell cat to use the same filename for both input and output, if you do you may fill up your hard drive very quickly as you try and create an infinite size file.

3. If you give a pattern like *.fasta and it doesn't match any files then the literal string *.fasta is used.

4. If you have a file called *.fasta then one way to tell the difference between that and the pattern is to see if it is readable.

5. Putting -- to end argument parsing is a good idea if there might be rogue filenames.

First a simple script.

# Simple script, assumes that "Species1" has all the needed "SequenceX.fasta" files
# Start in the directory containing "Species1", "Species2" etc.
# create output directory
mkdir "final"
# Go into the first directory
cd "Species1"
# Loop over all the files
for i in *".fasta"
do
    # join all the like named files in the sibling directories to the output
    # use a pattern which doesn't match ../final/$i to get list of files to join.
    cat "../Species"*"/$i" > "../final/$i"
done

This assumes that "Species1" has all the "SequenceX.fasta" files. If this isn't the case then you probably need a double loop. This is more robust but longer and slower.

# Start in the top level and loop over the directories
for dir in */
do
    # don't do anything inn the output directory
    [ "$dir" = "final" ] && continue
    # cd into directory, protecting against rogue directory names
    cd "./$dir" || { echo "cd to $dir failed" >&2 ; exit 1 ; }
    # loop over the files 
    for file in *"fasta"
    do
         # check the file exists, if there are no files matching the pattern
         # then the shell will pass the pattern through to the loop
         if [ -r "$file" ] ; then
             cat -- "$file" >> "../final/$file"
         fi
    done
    cd ".." || { echo "failed to return from $dir" ; exit 1 ; }
done