I have 81 files in .fasta format that contain (up to) 53 items. Such as:


Each .fasta file contains a name ID and string of characters delimited as:

>ATXGXXXXX(up to 53)

I would like to copy strings from each of my 81 .fasta files into a new .fasta file such that:


And the content of one of these contains (after copying from all 'Sample_X.fasta' files in the directory):


I've come across one or two similar questions, but nothing with exactly the nuance of preserving the SampleName in the copied output file and am having some difficulty getting examples from similar but different questions to work.

Thank you so much for any help!

3 Answers 3


I have the following code for you; below it there's an explanation how it works.

First go into the working directory (cd /User/MyData/) to run this program:

awk '
  FNR==1 { sample = FILENAME ; sub(/\.fasta/, "", sample }
  /^>/   { target = substr($0,2)".fasta" ; next }
         { print ">" sample > target ; print > target }
' Sample_*.fasta

The awk program iterates over all Sample_*.fasta files. At start of each input file (FNR==1) it extracts the sample name from the current filename by removing the suffix ".fasta". If a line starts with > then the target filename for that record is taken from after the > character, and the filename suffix ".fasta" appended. For the other type of lines the previously extracted sample name is written to the target file, and in a second line the current data is written.

Note: If you observe problems with "too many open file descriptors" then the best choice is to switch to GNU awk if possible!

If GNU awk is not or can not be made available on your platform then you need a couple of additional changes; the key is to close each file after writing to it, by using the close() function, with the consequence that you have to append to the closed files. (This is more complex and also less performant, so it's worth thinking about getting GNU awk and use the first variant.)

Those changes would then result in a program like:

# because of the append operation you need to empty the file targets
# before calling subsequent awk code, e.g. by: rm -f AT???????.fasta
awk '
  FNR==1 { sample = FILENAME ; sub(/\.fasta/, "", sample }
  /^>/   { target = substr($0,2)".fasta" ; next }
         { printf ">%s\n%s\n", sample, %0 >> target ; close(target) }
' Sample_*.fasta

Note that before you call the awk program you have to make sure to remove or empty any existing output-files from previous calls (otherwise your new output would get appended to the data previously existing in the respective output file(s).

  • Basically we came up with the same solution, although for the redirection >> should be used.
    – FloHimself
    Apr 27, 2015 at 6:30
  • 1
    @FloHinself; No, in awk it is sufficient and correct to write > as redirection to once create the file on start. (Using >> would lead to wrong results in case that the target file is already existing or if you run the program twice without removing the previous file.)
    – Janis
    Apr 27, 2015 at 6:34
  • Thanks for clarifying about the > operator, I didn't know it works this way in awk! Note the regex in the sub() call should use an anchor, otherwise a filename like Sample_1.fastasample.fasta will give strange results.
    – FloHimself
    Apr 27, 2015 at 7:16
  • @FloHimself; Thanks; the filenames are well specified in the question, so there's no need for that.
    – Janis
    Apr 27, 2015 at 7:26
  • Well this would leed to a potential issue if files will be added.
    – FloHimself
    Apr 27, 2015 at 7:27

Although it would be interesting to know what you've tried so far, here is an example how awk could be used for this job:

awk '
    FNR == 1 {
        sub(/\.fasta$/, "", FILENAME)
    /^>/ && sub(/^>/, "") {
        newfile = $0 ".fasta"
        print ">" FILENAME >> newfile
        print $0 >> newfile
    }' Sample_*.fasta
  • While similar to my solution this one has two caveats that should be noted. One is that it wrongly appends to the file >> instead of newly creating it; > should be used. (Mind that awk redirection works different than shell redirection!) And second, it is not optimal (but doesn't hurt in this case) to overwrite existing variables (FILENAME and $0); in case the script will be extended that may lead to surprises.
    – Janis
    Apr 27, 2015 at 6:37
  • Thanks for your comment, I didn't know that > worked this way in awk. So it depends on how the OP will use the script then. I thought overwriting build-in variables is a commonly used feature in awk as in $1 = ""; print $0 to print all but the first field...
    – FloHimself
    Apr 27, 2015 at 6:45
  • As said, the overwriting of those variables is only a potential issue if the script will be extended. I just wouldn't change predefined variables like FILENAME. Code like $1="" is even more a problem, because it changes the spacing of the input data and so $0 (i.e. in $0 the spaces between the fields), which you may not want reformatted. - WRT the redirection it's definitely > that should be used! (The >> does only open gaps for unnecessary problems, so avoid it here.) I suggest to fix that detail in your answer.
    – Janis
    Apr 27, 2015 at 6:52
  • I don't see a problem here in overwriting the build-ins, extensibility is cleary not an requirement here. With the change from >> to >, the script has to be called with all files given. A later call to the script will overwrite the previously written file.
    – FloHimself
    Apr 27, 2015 at 7:04
  • Even in case that a second call of the script will be made; your version will add duplicate records to the already created files. Where if using > you will always have a consistent result. - Anyway; keep it as you like. I noted the caveats and explained them, so folks will know what they get.
    – Janis
    Apr 27, 2015 at 7:14

Some shell: this will be much slower than an awk program.

cd /User/MyData
for sample in Sample*.fasta; do
    while read name; read data; do
        printf ">%s\n%s\n" "$sample_name" "$data" >> "$name.fasta"
    done < "$sample"
  • This also works! It's not noticeably slower for the size of files I'm working with, but I do encounter the same problem that I did above, with the awk program. Each "sample.fasta" produces a separate file for each data line "ATXGXXXXXX". What I need most is a single .fasta for each "ATXGXXXXX". Most samples contain the same set of named data (e.g., Samples 1 and 2 both contain a string of characters for AT1G000001 and AT2G000002. However, when I run these scripts for a folder with only two Sample_.fasta files I get two files named "AT1G000001" and two files names "AT2G000002".
    – MoGo
    Apr 27, 2015 at 20:26
  • @MoGo; For larger data shell loops are known to be slow. But (the observed discrepancy aside for a moment) depending on the magnitude of the degradation (i.e. if it's not too large) of the shell solution it may be worth to try running it in ksh (in case you tried withbash); there's a good chance that it's running significantly faster in ksh.
    – Janis
    Apr 27, 2015 at 20:40
  • I figured it out in my comment above. I don't know how I didn't think about impossibility of two same-named files in same directory... works great, thank you so much! I did a full export (I was just working with a small subset of two of the Sample files to test the scripts before). The awk script is much faster for the full file set!
    – MoGo
    Apr 27, 2015 at 20:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .