How to copy lines from multiple files into one new file and keep file name?

Question

I have 81 files in .fasta format that contain (up to) 53 items. Such as:

/User/MyData/Sample_1.fasta
/User/MyData/Sample_2.fasta
....
/User/MyData/Sample_81.fasta

Each .fasta file contains a name ID and string of characters delimited as:

>AT1G00001
ATCCACTGCTGTGTACCTGATCAGTGCTGACCCAYTGTGACACTGTG
>AT2G00002
AAAAATTTTGCCCGTGTGGGCCAAACTGTCATGCATGCACCGTACGTGCATGCAT
....
>ATXGXXXXX(up to 53)
AAACCCTCTTTGTGCCTGTGCATGCA

I would like to copy strings from each of my 81 .fasta files into a new .fasta file such that:

/User/MyData/AT1G00001.fasta
/User/MyData/AT2G00002.fasta
....
/User/MyData/ATXGXXXXX.fasta

And the content of one of these contains (after copying from all 'Sample_X.fasta' files in the directory):

>Sample_1
ATCCACTGCTGTGTACCTGATCAGTGCTGACCCAYTGTGACACTGTG
>Sample_2
ATCGACTCCCGTAGGACTGATTTTTCTGACCCCATTGTGACACTGTG
....
>Sample_81
TTCTGACCCCATTGTGACACTGTGATCGACTCCCGTAGGACTGATTT

I've come across one or two similar questions, but nothing with exactly the nuance of preserving the SampleName in the copied output file and am having some difficulty getting examples from similar but different questions to work.

Thank you so much for any help!

Answer 1

I have the following code for you; below it there's an explanation how it works.

First go into the working directory (cd /User/MyData/) to run this program:

awk '
  FNR==1 { sample = FILENAME ; sub(/\.fasta/, "", sample }
  /^>/   { target = substr($0,2)".fasta" ; next }
         { print ">" sample > target ; print > target }
' Sample_*.fasta

The awk program iterates over all Sample_*.fasta files. At start of each input file (FNR==1) it extracts the sample name from the current filename by removing the suffix ".fasta". If a line starts with > then the target filename for that record is taken from after the > character, and the filename suffix ".fasta" appended. For the other type of lines the previously extracted sample name is written to the target file, and in a second line the current data is written.

Note: If you observe problems with "too many open file descriptors" then the best choice is to switch to GNU awk if possible!

If GNU awk is not or can not be made available on your platform then you need a couple of additional changes; the key is to close each file after writing to it, by using the close() function, with the consequence that you have to append to the closed files. (This is more complex and also less performant, so it's worth thinking about getting GNU awk and use the first variant.)

Those changes would then result in a program like:

# because of the append operation you need to empty the file targets
# before calling subsequent awk code, e.g. by: rm -f AT???????.fasta
awk '
  FNR==1 { sample = FILENAME ; sub(/\.fasta/, "", sample }
  /^>/   { target = substr($0,2)".fasta" ; next }
         { printf ">%s\n%s\n", sample, %0 >> target ; close(target) }
' Sample_*.fasta

Note that before you call the awk program you have to make sure to remove or empty any existing output-files from previous calls (otherwise your new output would get appended to the data previously existing in the respective output file(s).

Answer 2

Although it would be interesting to know what you've tried so far, here is an example how awk could be used for this job:

awk '
    FNR == 1 {
        sub(/\.fasta$/, "", FILENAME)
    }
    /^>/ && sub(/^>/, "") {
        newfile = $0 ".fasta"
        next
    }
    {
        print ">" FILENAME >> newfile
        print $0 >> newfile
    }' Sample_*.fasta

Answer 3

Some shell: this will be much slower than an awk program.

cd /User/MyData
for sample in Sample*.fasta; do
    sample_name=${sample%.fasta}
    while read name; read data; do
        name=${name#>}
        printf ">%s\n%s\n" "$sample_name" "$data" >> "$name.fasta"
    done < "$sample"
done