0

I have a series of strings. These are composed of smaller strings labeled as 'nodes', sometimes on their own, sometimes connected by characters : or ,.

I'd like to split the larger strings ('headers') into 'nodes'.

I've managed to remove some extra characters (>, ;, ') with sed, and I'm using awk to split the remaining strings on : and ,

The issue is that I'd like to loop through the output (the 'nodes'), not only the first awk column. I've tried using {print $0} for awk, but that just printed out the initial strings with delimiters and all.

Please help?

Example input (dealt with by a for loop in the sample, in the larger code it's an output of an if/else):

>NODE_3028138_length_2215_cov_1.9513_ID_6056275:NODE_6264558_length_375_cov_4.0000_ID_12529115';
>NODE_4338305_length_1150_cov_1.0000_ID_8676609;
>NODE_3552704_length_509_cov_1.0000_ID_7105407:NODE_4456634_length_439_cov_1.9597_ID_8913267',NODE_4457268_length_491_cov_0.9657_ID_8914535';

Example output (without node NODE_4338305 because it's on its own):

NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535

Ideally I would want to loop through each of the entries above (NODE_3028138_length_2215_cov_1.9513_ID_6056275, then NODE_6264558_length_375_cov_4.0000_ID_12529115 etc)

for i in ">NODE_3028138_length_2215_cov_1.9513_ID_6056275:NODE_6264558_length_375_cov_4.0000_ID_12529115';" \
">NODE_4338305_length_1150_cov_1.0000_ID_8676609;" \
">NODE_3552704_length_509_cov_1.0000_ID_7105407:NODE_4456634_length_439_cov_1.9597_ID_8913267',NODE_4457268_length_491_cov_0.9657_ID_8914535';"
do      
if [[ $i == *":"* ]];         
then 
echo $i            
i=$(sed "s/[>;\']//g" <<< $i);            
echo $i
echo $i | awk -F '[:,]' '{print $1}' | while IFS= read -r line; do echo "$line"; done
fi; done

Edit to add operating system information:

  • Operating System: CentOS Linux 7 (Core)
  • Kernel: Linux 3.10.0-1127.el7.x86_64
  • Architecture: x86-64
11
  • 1
    To loop through fields in awk: {for (i=1; i<=NF; ++i) {print $i}}. But it is unclear to me if this is the answer to your question.
    – guest
    Jul 9, 2020 at 12:17
  • 1
    Since you're already using sed, why not add a second expression that replaces all [:,] with newlines to that? Jul 9, 2020 at 12:20
  • @steeldriver Thank you! The idea sounds very intuitive, but I've tried 'i=$(sed "s/[:,]/\n/g" <<< $i)' and only got spaces, not new lines, and I also tried 'i=$(sed "s/[:,]/\r/g" <<< $i)' and...I only got the last node, so I'm not sure what went wrong.
    – Laura
    Jul 9, 2020 at 12:29
  • @guest Thank you, this works well!
    – Laura
    Jul 9, 2020 at 12:32
  • 5
    @Laura please edit your question and tell us what operating system you are using. Your comment about sed suggests you are not using GNU sed. Also, please clarify what you want to loop over because it isn't clear at all. If you already have awk there why would you want a shell loop at all? Just show us your input, and your desired output. We should be able to give you a far simpler approach than what you're using.
    – terdon
    Jul 9, 2020 at 12:51

5 Answers 5

3

You don't need any of the steps you show. If I understand correctly, you start with a collection of fasta files, a format that looks like this:

>header
sequence

And you want to extract the headers, remove the > and any ' and split them on , or ;. If so, you can do this directly on the fasta files themselves:

$ sed -n '/^>/{s/>//; s/[,:]/\n/gp}' *.fasta | tr -d "';"
NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535

Explanation

  • sed -n: suppress normal output, don't print anything unless explicitly told to.
  • /^>/{something}: if this line starts with a >, do something.
  • s/^>//;: remove the > from the beginning of the line.
  • s/[,:]/\n/gp : replace all (all because of the g at the end) , or : with a newline (\n) and then print (you print because of the p at the end.
  • tr -d "';": delete any ; or '.

In your comment, you said you tried 'i=$(sed "s/[:,]/\n/g" <<< $i)' and only got spaces, not newlines. That's because you then ran echo $i instead of echo "$i", so the newlines were lost.


If you really do need to do this on the collection of strings you show, you can do:

for i in ">NODE_3028138_length_2215_cov_1.9513_ID_6056275:NODE_6264558_length_375_cov_4.0000_ID_12529115';" ">NODE_4338305_length_1150_cov_1.0000_ID_8676609;" ">NODE_3552704_length_509_cov_1.0000_ID_7105407:NODE_4456634_length_439_cov_1.9597_ID_8913267',NODE_4457268_length_491_cov_0.9657_ID_8914535';"; do 
    sed -n '/^>/{s/>//; s/[,:]/\n/gp}' <<<"$i" | tr -d "';" ; 
done
NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535
2

The following solution is somewhat crude, but should work. It is assumed that, as you show in your sample input, all nodes begin with the string NODE (if that is not the case, you need to provide a more complete input example).

Assuming your strings are, in fact, in a file input.txt, the following awk call will do the trick:

awk '{gsub(/[:>,;\047]/,""); n=split($0,a,/NODE/); for (i=2;i<=n;i++) printf("NODE%s\n",a[i])}' input.txt
  • This will first replace all "extra" characters using gsub() (\047 is the single quote ' which can't be placed verbatim on the command-line as the awk commands themselves are inside single-quotes).
  • Then, it will split the remaining string into fields at the pattern NODE, and store the result in the array a.
  • Any "field", except for the very first (which would be the string before the first occurence of NODE) is then printed separately, with NODE being prepended.

For your example input the result is:

awk '{gsub(/[:>,;\047]/,""); n=split($0,a,/NODE/); for (i=2;i<=n;i++) printf("NODE%s\n",a[i])}' input.txt
NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_4338305_length_1150_cov_1.0000_ID_8676609
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535

If you want to skip lines that only contain one such "node", the command can be modified as:

awk '{gsub(/[:>,;\047]/,""); if ((n=split($0,a,/NODE/))<3) next; for (i=2;i<=n;i++) printf("NODE%s\n",a[i])}' input.txt
NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535
1
  • 1
    @EdMorton good point; included!
    – AdminBee
    Jul 10, 2020 at 12:18
0

Tried with below python method

#!/usr/bin/python
import re
m=re.compile(r'[:;,]')
k=open('filename','r')
for i in k:
    co=i.count("NODE")
    if co > 1:
        q=i.strip()
        k=re.sub(m,"\n",q)
        print k.strip().replace("'","").replace(">","")

output

NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535

awk: Already best solution provided in awk its just my try

awk '{print $0,gsub("NODE",$0)}' filename| awk '$NF >1 {print $1}'| sed "s/[;:,]/\n/g"|sed '/^$/d'| sed "s/[\>']//g"
0

If you have gawk and don't mind a leading blank line then you can

  • compose RS to consume '*;\n which cleans up the end of each line
  • compose FS to consume combinations of [>;,']+
  • compose OFS to provide \n and
  • set ORS to empty.

This means that the empty $1 (the field before >) gives you a \n between records if you recompose $0 on OFS with $1=$1. Other than that just print wherever you have NF>2

awk -F"[>:,']+" -v OFS="\n" -v RS="'*;\n" -v ORS="" 'NF>2{$1=$1; print }END{print "\n"}' file

NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535 
0

Using the sed editor we can generate the desired output as follows.

sed \
  -e '/\n/{/^\n/!P;D;}'                    \
  -e "/^>NODE_.*NODE/ y/>;:,'/\n\n\n\n\n/" \
  -e '/\n/G;D'                             \
file

Results:

NODE_3028138_length_2215_cov_1.9513_ID_6056275
NODE_6264558_length_375_cov_4.0000_ID_12529115
NODE_3552704_length_509_cov_1.0000_ID_7105407
NODE_4456634_length_439_cov_1.9597_ID_8913267
NODE_4457268_length_491_cov_0.9657_ID_8914535

Working method:

  • Only lines that comprise atleast two NODEs plus they begin with >NODE_ Let’s call them "interesting" lines. We change each occurrence of >;:,' to a newline each in the interesting lines.
  • Then append a newline to the interesting lines just in case it didn't end with a semicolon. The D command shall initiate an implicit looping and take us to the first line of sed code.
  • The first line is where all the action happens and the interesting line is consumed fully while sed belts out the NODEs serially, one per line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.