0

Can anyone help me with the following issue? I have circa 40 directories of different species, each with 100s of sequence files that contain orthologous sequences. The sequence files are similarly named for each of the species directories. I want to concatenate the identically named files of the 40 species directories into a single sequence file which is named similarly.

E.g. I have the following 3 directories: "Species1", "Species2", "Species3". Within these directories are similarly named files: "SequenceA.fasta", "SequenceB.fasta", "SequenceC.fasta". I need to concatenate all contents of the different SequenceA.fasta files into one new file named "SequenceA.fasta" in another directory. How do I do this?

I tried it with the following loop, which failed. Files are created but are empty:

ls . | while read FILE; do cat ./*/"$FILE" >> ./final/"$FILE"; done

Thanks for any advice or help!

(Sorry for any potential cross-posting, I accidentally posted this question in a wrong forum earlier)

0

There are several things to note in this answer.

  1. It is generally a bad idea to parse the output of ls if you can instead use shell glob patterns to do what you want instead - see [http://mywiki.wooledge.org/ParsingLs]. I didn't use nullglob shell option for portability, but that would make the scripts slightly shorter.

  2. You want to make sure that your glob patterns are not too broad so you tell cat to use the same filename for both input and output, if you do you may fill up your hard drive very quickly as you try and create an infinite size file.

  3. If you give a pattern like *.fasta and it doesn't match any files then the literal string *.fasta is used.

  4. If you have a file called *.fasta then one way to tell the difference between that and the pattern is to see if it is readable.

  5. Putting -- to end argument parsing is a good idea if there might be rogue filenames.

First a simple script.

# Simple script, assumes that "Species1" has all the needed "SequenceX.fasta" files
# Start in the directory containing "Species1", "Species2" etc.
# create output directory
mkdir "final"
# Go into the first directory
cd "Species1"
# Loop over all the files
for i in *".fasta"
do
    # join all the like named files in the sibling directories to the output
    # use a pattern which doesn't match ../final/$i to get list of files to join.
    cat "../Species"*"/$i" > "../final/$i"
done

This assumes that "Species1" has all the "SequenceX.fasta" files. If this isn't the case then you probably need a double loop. This is more robust but longer and slower.

# Start in the top level and loop over the directories
for dir in */
do
    # don't do anything inn the output directory
    [ "$dir" = "final" ] && continue
    # cd into directory, protecting against rogue directory names
    cd "./$dir" || { echo "cd to $dir failed" >&2 ; exit 1 ; }
    # loop over the files 
    for file in *"fasta"
    do
         # check the file exists, if there are no files matching the pattern
         # then the shell will pass the pattern through to the loop
         if [ -r "$file" ] ; then
             cat -- "$file" >> "../final/$file"
         fi
    done
    cd ".." || { echo "failed to return from $dir" ; exit 1 ; }
done

3
  • Thanks so much! Your answer is extremely helpful for me (as someone that needs to learn still so much)! Because 'Species1' does indeed not have all the necessary files I have used the second script and it seems to work as a charm! However, I did get the following output (which I not get implementing it on a small test dataset): cat: 99964at50557.faa.fasta: input file is output file I do not really get what it means?
    – TUnix
    Apr 2 '20 at 12:45
  • GNU's version of cat is smart enough to detect that the input and output files are the same and print this message and stop. This is a problem because if say the file contains this is a dog, and you append the file to itself you get this is a dogthis is a dog. At this point cat sees there is more text in the file so it adds it, and the file is now this is a dogthis is a dogthis is a dog and now there is more text and that gets added so the file is now this is a dogthis is a dogthis is a dogthis is a dog and so on until the disk is full. This is why point 2 is important. cont...
    – icarus
    Apr 2 '20 at 15:45
  • ... The second script took care (I hope) to make sure it didn't do this by not going into the directory with the results with the [ "$dir" = "final" ] && continue line.
    – icarus
    Apr 2 '20 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.