I have 81 files in .fasta format that contain (up to) 53 items. Such as:


Each .fasta file contains a name ID and string of characters delimited as:

>ATXGXXXXX(up to 53)

I would like to copy strings from each of my 81 .fasta files into a new .fasta file such that:


And the content of one of these contains (after copying from all 'Sample_X.fasta' files in the directory):


I've come across one or two similar questions, but nothing with exactly the nuance of preserving the SampleName in the copied output file and am having some difficulty getting examples from similar but different questions to work.

Thank you so much for any help!


I have the following code for you; below it there's an explanation how it works.

First go into the working directory (cd /User/MyData/) to run this program:

awk '
  FNR==1 { sample = FILENAME ; sub(/\.fasta/, "", sample }
  /^>/   { target = substr($0,2)".fasta" ; next }
         { print ">" sample > target ; print > target }
' Sample_*.fasta

The awk program iterates over all Sample_*.fasta files. At start of each input file (FNR==1) it extracts the sample name from the current filename by removing the suffix ".fasta". If a line starts with > then the target filename for that record is taken from after the > character, and the filename suffix ".fasta" appended. For the other type of lines the previously extracted sample name is written to the target file, and in a second line the current data is written.

Note: If you observe problems with "too many open file descriptors" then the best choice is to switch to GNU awk if possible!

If GNU awk is not or can not be made available on your platform then you need a couple of additional changes; the key is to close each file after writing to it, by using the close() function, with the consequence that you have to append to the closed files. (This is more complex and also less performant, so it's worth thinking about getting GNU awk and use the first variant.)

Those changes would then result in a program like:

# because of the append operation you need to empty the file targets
# before calling subsequent awk code, e.g. by: rm -f AT???????.fasta
awk '
  FNR==1 { sample = FILENAME ; sub(/\.fasta/, "", sample }
  /^>/   { target = substr($0,2)".fasta" ; next }
         { printf ">%s\n%s\n", sample, %0 >> target ; close(target) }
' Sample_*.fasta

Note that before you call the awk program you have to make sure to remove or empty any existing output-files from previous calls (otherwise your new output would get appended to the data previously existing in the respective output file(s).

| improve this answer | |
  • Basically we came up with the same solution, although for the redirection >> should be used. – FloHimself Apr 27 '15 at 6:30
  • 1
    @FloHinself; No, in awk it is sufficient and correct to write > as redirection to once create the file on start. (Using >> would lead to wrong results in case that the target file is already existing or if you run the program twice without removing the previous file.) – Janis Apr 27 '15 at 6:34
  • Thanks for clarifying about the > operator, I didn't know it works this way in awk! Note the regex in the sub() call should use an anchor, otherwise a filename like Sample_1.fastasample.fasta will give strange results. – FloHimself Apr 27 '15 at 7:16
  • @FloHimself; Thanks; the filenames are well specified in the question, so there's no need for that. – Janis Apr 27 '15 at 7:26
  • Well this would leed to a potential issue if files will be added. – FloHimself Apr 27 '15 at 7:27

Although it would be interesting to know what you've tried so far, here is an example how awk could be used for this job:

awk '
    FNR == 1 {
        sub(/\.fasta$/, "", FILENAME)
    /^>/ && sub(/^>/, "") {
        newfile = $0 ".fasta"
        print ">" FILENAME >> newfile
        print $0 >> newfile
    }' Sample_*.fasta
| improve this answer | |
  • While similar to my solution this one has two caveats that should be noted. One is that it wrongly appends to the file >> instead of newly creating it; > should be used. (Mind that awk redirection works different than shell redirection!) And second, it is not optimal (but doesn't hurt in this case) to overwrite existing variables (FILENAME and $0); in case the script will be extended that may lead to surprises. – Janis Apr 27 '15 at 6:37
  • Thanks for your comment, I didn't know that > worked this way in awk. So it depends on how the OP will use the script then. I thought overwriting build-in variables is a commonly used feature in awk as in $1 = ""; print $0 to print all but the first field... – FloHimself Apr 27 '15 at 6:45
  • As said, the overwriting of those variables is only a potential issue if the script will be extended. I just wouldn't change predefined variables like FILENAME. Code like $1="" is even more a problem, because it changes the spacing of the input data and so $0 (i.e. in $0 the spaces between the fields), which you may not want reformatted. - WRT the redirection it's definitely > that should be used! (The >> does only open gaps for unnecessary problems, so avoid it here.) I suggest to fix that detail in your answer. – Janis Apr 27 '15 at 6:52
  • I don't see a problem here in overwriting the build-ins, extensibility is cleary not an requirement here. With the change from >> to >, the script has to be called with all files given. A later call to the script will overwrite the previously written file. – FloHimself Apr 27 '15 at 7:04
  • Even in case that a second call of the script will be made; your version will add duplicate records to the already created files. Where if using > you will always have a consistent result. - Anyway; keep it as you like. I noted the caveats and explained them, so folks will know what they get. – Janis Apr 27 '15 at 7:14

Some shell: this will be much slower than an awk program.

cd /User/MyData
for sample in Sample*.fasta; do
    while read name; read data; do
        printf ">%s\n%s\n" "$sample_name" "$data" >> "$name.fasta"
    done < "$sample"
| improve this answer | |
  • This also works! It's not noticeably slower for the size of files I'm working with, but I do encounter the same problem that I did above, with the awk program. Each "sample.fasta" produces a separate file for each data line "ATXGXXXXXX". What I need most is a single .fasta for each "ATXGXXXXX". Most samples contain the same set of named data (e.g., Samples 1 and 2 both contain a string of characters for AT1G000001 and AT2G000002. However, when I run these scripts for a folder with only two Sample_.fasta files I get two files named "AT1G000001" and two files names "AT2G000002". – MoGo Apr 27 '15 at 20:26
  • @MoGo; For larger data shell loops are known to be slow. But (the observed discrepancy aside for a moment) depending on the magnitude of the degradation (i.e. if it's not too large) of the shell solution it may be worth to try running it in ksh (in case you tried withbash); there's a good chance that it's running significantly faster in ksh. – Janis Apr 27 '15 at 20:40
  • I figured it out in my comment above. I don't know how I didn't think about impossibility of two same-named files in same directory... works great, thank you so much! I did a full export (I was just working with a small subset of two of the Sample files to test the scripts before). The awk script is much faster for the full file set! – MoGo Apr 27 '15 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.