0

My question is: I have 200 files in fata format. Such as:

 /User/Bin/bin.0.fa
 /User/Bin/bin.1.fa
 ...
 /User/Bin/bin.200.fa

Each .fa file contains a contig name ID and nucleotide characters delimited as:

In /User/Bin/bin.0.fa

>c_000000000001
CGACATTTTCCAACTTATTTTTTCCTGTAGTAAAAATTATTTACATACAAAAAAGGAGCTGTTCACTAATTATTTAGTGC
>c_000000000002 
TACAACTCCTTTTTACTATTCTTCTGAATTTGATTTTTCATCCATTTGTTTTTGAGCTTCTTGAACTAATTTATCAAGACTATTATCTTCTACAACTTCATTTTCTTGTCTATCTAATTCATCTGTTAATGTTAATTGCTGATCTTTATCTTCTACATCT CTACCTGAAATTTTAGCTATAGCTACAATCTTTTCTTCATCAGAAGTTCTCATTAATCTAACTCCCATTGTAGCTCTAC
>c_000000000003  
AGTTACAGATACATCTGATACATTAATTCTTATAGCAACACCACTTGTATTTATAAGCATTAATTCATCTTCAGATTTACATACTGTTGCACCAACAACTTTACCAGTCTTTTCACTGATTTTGTATGTTATTAAACCAACTCCACCTCTATTTTGTCTC
...

In /User/Bin/bin.1.fa

>c_000000000004
GGATCATCGCTTGTACATCCCAAACCAAAAAAGAATACTGCACTTACAATCAGTTGGATTTGAAACGCGATTTTCATTTTTGGTATATGTTTAAGATTAGCACTTTGTTTCATTGCTTTTGGCTATGAACGATGTTTACGGGGGTGTA
>c_000000000005 
GAAAGAAGCGTATTGGTCGGTATAAATACCGCTCAACTAAACGAGCACAAAGCTACCGAAAATTTGGATGAATTGGCTTTTCTGGCCCAAACGGCTGGAGC
>c_000000000006
CGGCACTTATTTGCCCCAGCCCATTTTGGGGGTAGAAATACCCAAGAGCAAGGGAAAGGTTCGCCTTCTGGGTGTGCCTACCGTGGTTGACCGTATGTTGCAAC
... 

...
In /User/Bin/bin.200.fa

>c_000000020120   
CTCTGCAACTGGATCCCGAAAAGATCCGCAAAGAAAGCGAACCCAAAGAAAAAGTCGATCTGGAGAGCACCGTCGCCCGCAGTCTGGCCACCCT
>c_000000020121
CATCAATCATCTCAAATACTACCGCAACGCAGATTATTCCCAGTGCAATAACAAAACCGACTCCCGCCTCTTTTGTCTGGCCGTA
>c_000000050122 
GGTACGCCTCCGGCAGAACAAGGCGGCAACGAACCTCAGAACGAGGGAAAGCTAACCCAGGCCGGGTACGCCTCCGGCAGAACAAGGCGGCAACGAACCTCAGAACGAGGGAAAGCTAACCCAGGCCG
...  

I would like to copy each contig name ID (without ">") from specific .fa files into a single TAB delimited txt file where the contig name IDs are named as the originated file n+1. Like this:

In /User/Bin/Summary.txt

c_000000000001 Bin_1
c_000000000002 Bin_1
c_000000000003 Bin_1
...
c_000000000004 Bin_2
c_000000000005 Bin_2
c_000000000006 Bin_2
...
...
c_000000020120 Bin_201
c_000000020121 Bin_201
c_000000020122 Bin_201
2
  • Are >c_000.. and CTGATAC on the same line or is the ID a separate line?
    – FelixJN
    Commented Jan 10, 2020 at 9:33
  • Like in the example: >c_0000... CTGATAC The sequence nucleotide caracters are in separate line.
    – Roland
    Commented Jan 10, 2020 at 9:37

6 Answers 6

1

Given your posted sample input/output and the answer you accepted, all you actually need is this using GNU awk for ARGIND:

awk -F'>' -v OFS='\t' 'NF>1{print $2, "Bin_"ARGIND}' /Usr/Bin/bin*.fa > /User/Bin/Summary.txt

or with any awk:

awk -F'>' -v OFS='\t' 'FNR==1{++c} NF>1{print $2, "Bin_"c}' /Usr/Bin/bin*.fa > /User/Bin/Summary.txt
2
  • That is a very elegant approach. However, it relies on there being a continuous numbering of the input filenames without gaps, which in practice may not be the case (even in the example, where the OP stated "I have 200 files", but then gives bin.0.fa ... bin.200.fa as filenames; although that may of course have been a typo). Then again, it is only my interpretation of the OP's statement "named as the original file n+1" that he wanted the ID in the output file to be tied to the input filname, and your answer may be just as correct.
    – AdminBee
    Commented Jan 13, 2020 at 8:06
  • 1
    Yes indeed! These commands also works. Thank you!
    – Roland
    Commented Jan 13, 2020 at 8:14
0
#!/usr/bin/env python

import os

files = os.listdir('/User/Bin')
for file in files:
    fi = open(file, 'r')
    n = file.split('.')[1]
    for line in fi:
        line = line.strip()
        if line.startswith('>'):
            bins = 'Bin_' + n
            print("%s\t%s" % (line[1:], bins))
    fi.close()

As long as you are on Linux, you might have python installed. This might do the trick.

1
  • Yes it is a nother solution for the problem. Thank you!
    – Roland
    Commented Jan 10, 2020 at 10:08
0

The command that could accomplish this is as follows

grep "^>" /User/Bin/*.fa | awk -F ":>|\\." '{sub(".*/", "", $1);printf "%s\t%s_%d\n",$4,$1,$2+1}'

Let me explain the various steps

grep "^>" /User/Bin/*.fa

the above command greps only the lines with the the required pattern and prints the file names along with the line. We could have used only awk to print matching lines, but by using grep we make it easier for us to do the operations you require on the file name.

awk -F ":>|\\." 

This makes the ":>" characters or "." as the field separator , thus splitting the output of the grep in 4 parts - 1. file name till the first "." 2. index name 3. Extension 4. Actual pattern grepped

{sub(".*/", "", $1)

The above step removes the basename from the file name in the first part.

printf "%s\t%s.%d\n",$4,$1,$2+1}

does the final printing where the second part is incremented by 1 , as you require it and a tab is inserted for tab separation.

1
  • Nice workflow. Thank you for the explanations.
    – Roland
    Commented Jan 10, 2020 at 10:10
0

Try also

awk 'FNR == 1 {split (FILENAME, T, "."); FN = T[1] "_" ++T[2]}; /^>/ {print substr ($1, 2), FN}' *.fa
c_000000000001 bin_1
c_000000000002 bin_1
c_000000000003 bin_1
c_000000000004 bin_2
c_000000000005 bin_2
c_000000000006 bin_2
c_000000020120 bin_201
c_000000020121 bin_201
c_000000050122 bin_201
0

Alternative awk grep combinations

grep -Eo "c_[0-9]+" *.fa | awk -F'[.:]' '{print $4,"Bin_"$2+1}' | column -t

or

grep -Eo "c_[0-9]+" *.fa | awk -F'[.:]' 'BEGIN{OFS="\t"}{print $4,"Bin_"$2+1}'

or if it's not always "bin...."

grep -Eo "c_[0-9]+" *.fa | awk -F'[.:]' 'BEGIN{OFS="\t"}{print $4,toupper(substr($1,1,1))substr($1,2)"_"$2+1}'

Note that this use of grep will fail if there is only 1 input file, in which case you need to use the -H option

6
  • Thanks for the catch
    – bu5hman
    Commented Jan 10, 2020 at 13:50
  • You never need grep when you're using awk.
    – Ed Morton
    Commented Jan 10, 2020 at 16:19
  • Indeed it's true you don't, but I was too lazy to type all of @AdminBee solution and it won't hurt OP to see alternative ideas and tool usage. Perhaps a but golfy too.
    – bu5hman
    Commented Jan 10, 2020 at 16:36
  • 1
    You shoulld mention it requires GNU grep for its experimental -P option or just change -P to -E since you don't actually require any PCRE functionality in your regexp, just ERE for the +. Or even replace + with [0-9]* and then you don't need either.
    – Ed Morton
    Commented Jan 10, 2020 at 16:40
  • Fair enough on the -E but OP pattern requires a number after the c_ so its either [0-9]+ or [0-9][0-9]*
    – bu5hman
    Commented Jan 10, 2020 at 16:44
0

This should be possible with GNU Awk:

awk 'match($0,/^>([^[:space:]]+)/,v) {match(FILENAME,/^.*bin\.([[:digit:]]+)\.fa$/,fid); printf("%s\tBin_%s\n", v[1], fid[1]+1);}' /User/Bin/bin.*.fa > summary.txt

This will:

  • match all lines starting with > and extract the pattern behind the > character (up to the first whitespace) into the awk array variable v by use of the match() function in the GNU awk implementation
  • do the same to extract the filename number from the currently processed filename (stored in the internal variable FILENAME) to an array variable fid
  • print the contig ID (first element of v) and file id (Bin_ followed by the incremented value of fid[1])

The output of the awk call is then redirected into a file summary.txt

Edit

As suggested by Ed Morton, the efficiency can be improved by extracting the ID only on "change of input file", which can be detected by FNR (the per-file line number) being 1 (while still avoiding possible problems when there are gaps in the filename numbering).

Also, the use of match() can be avoided by trimming the line containing the "contig ID" from the leading > and any trailing whitespace; this also makes the code more portable across awk implementations.

awk 'FNR==1{n=split(FILENAME,fid,"[/.]"); out=fid[n-1]+1} index($0,">")==1{gsub(/(^>|[ \t]*$)/,""); printf("%s\tBin_%s\n", $0, out);}' /User/Bin/bin.*.fa
4
  • Thank you! Worked perfectly. :)
    – Roland
    Commented Jan 10, 2020 at 10:06
  • FWIW that same approach written more portably (it'll work in all awks) and efficiently (it only runs match() on the FILENAME once per input file rather than once per input line) would be (untested) awk 'FNR==1{n=split(FILENAME,fid,"[/.]"); out=fid[n-1]+1} sub(/^>/,"") {printf "%s\tBin_%s\n", $0, out}' /User/Bin/bin.*.fa > summary.txt
    – Ed Morton
    Commented Jan 10, 2020 at 16:37
  • 1
    @EdMorton that is a good idea to make it more efficient, and I shall include it in the answer. However, note that there seem to be trailing spaces and (possibly) horizontal tabs on the "contig ID" lines in the OPs input file examples, and considering these are probably directly copy-and-pasted from the original files, I thought it more "robust" to use the (admittedly expensive) match() call to extract the "contig ID" field.
    – AdminBee
    Commented Jan 13, 2020 at 8:31
  • Not sure the OP has or cares about that but if they do you could just use $1 instead of $0 when printing the config id since they don't appear to contain white space in their value.
    – Ed Morton
    Commented Jan 13, 2020 at 13:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .