2

I would like to replace number between patterns with multiplicated numbers and print the all the lines. The file is a tree file in newick format and consisted only a single line. My targets are all the numbers after ) and before :. I wanted to multiply all the numbers in between the two symbols with 100.

file:

((((A_8:0.000846,(A_5:0.002449,(A_1:1e-06,((A_4:1e-06,((A_7:1e-06,A_6:0.001061)0.714000:1e-06,A_3:1e-06)0.314500:1e-06)0.358667:1e-06,A_2:1e-06)0.361000:1e-06)0.434800:1e-06)0.683500:0.001619)0.888571:0.001931,A_9:0.00069)0.688471:0.000691,...

The easiest way to me seemed to be splitting the file by replacing all the ":" symbols with a new line first. So all my target numbers are now in separate lines and appear after ). Then, I was using the awk script below to multiply the target numbers with 100, but didn't manage to keep the lines without my target number though.

script:

sed 's/:/\n/g' df9.tree | awk -F")" '{OFS=")"} $2=$2*100 {print $0}'
sed 's/:/\n/g' df9.tree | awk '$NF ~/)/ {$NF *=100}1'

How can I multiply the numbers after ) and print the entire file in this case? Or is there other simpler way to directly look for the numbers lie between : and ), multiply them by 100 and print the whole file?

Update: Expected output

((((A_8:0.000846,(A_5:0.002449,(A_1:1e-06,((A_4:1e-06,((A_7:1e-06,A_6:0.001061)71.4000:1e-06,A_3:1e-06)31.4500:1e-06)35.8667:1e-06,A_2:1e-06)36.1000:1e-06)43.4800:1e-06)68.3500:0.001619)88.8571:0.001931,A_9:0.00069)68.8471:0.000691,...)
2
  • If you can describe the positions of the numbers to edit as parts of the tree, writing a your-favorite-lisp program to do this would both be easy and more robust. Jul 8, 2021 at 22:52
  • The most important part of your question is what the value of those ...s are because the right answer will depend on whether your input file ends with ) alone or )\n or ends in something else. So please replace the ...s with representative values as they appear in your real input file and if there is no newline at the end then tell us that.
    – Ed Morton
    Jul 9, 2021 at 12:04

4 Answers 4

6
awk 'BEGIN {OFS=FS=":"; ORS=RS=")"} NR>1 {$1=sprintf("%.4f", $1 * 100)}1' df9.tree

If you accept to separate RS records and FS fields, then the required number will always be in the first field after the first record.

6
  • 1
    Hi there, thanks for the answer. I tried the script on my file but it seemed that only work on the portion of the file I posted. For the rest of the files, all the : were being replaced to a space.
    – web
    Jul 8, 2021 at 13:53
  • 2
    try adding a space and -v OFS=: immediately after -F:. That will set the output field separator (OFS) to :, same as the input field separator (FS).
    – cas
    Jul 8, 2021 at 13:54
  • @cas, thanks. Fixed
    – nezabudka
    Jul 8, 2021 at 14:06
  • 1
    Nice! The only problem with it is that if the input doesn't end in )\n that'll add a ) to the output, and if the input does and with )\n then it'll add 0.0000 to the end of the output as it'll multiply the \n and in either case it won't provide a terminating newline and so the output isn't a valid POSIX text file and so YMMV with what any subsequent tool can do with it. If the input ends with ) with no terminating newline (i.e. isn't a valid POSIX text file) then what any text processing tool does with that is undefined behavior.
    – Ed Morton
    Jul 9, 2021 at 12:00
  • 1
    I don;t remember this question or answer as it's been a month and I'm not going to reread it now but one point about your comment - always use printf "%s", $0 instead of print $0 as the latter will fail when the input contains printf formatting characters like %s.
    – Ed Morton
    Aug 5, 2021 at 14:09
5
$ perl -pe 's/\)([-0-9.]+):/sprintf ")%.4f:", $1 * 100/eg' df9.tree
((((A_8:0.000846,(A_5:0.002449,(A_1:1e-06,((A:1e-06,((A_7:1e-06,A:0.001061)71.4000:1e-06,A:1e-06)31.4500:1e-06)35.8667:1e-06,A:1e-06)36.1000:1e-06)43.4800:1e-06)68.3500:0.001619)88.8571:0.001931,A:0.00069)68.8471:0.000691,...

replaces all numbers (defined as a sequence of one-or-more digits, periods, or minus characters) immediately following a ) character and terminated by a : character with the number multiplied by 100.

e.g. )0.714000: gets changed to )71.4000:

It uses perl's /e regex evaluation modifier to execute perl code in the RHS of the s/// operator. See man perlop and search for s\/PATTERN for details. sprintf is used to format the number to have 4 decimal places.


If the number between ) and : could be in either plain decimal notation ("0.714000") or "C float"-style scientific notation ("1e-06"), the regex needs to be just a tiny bit more complicated to match all the possible variations:

$ perl -pe 's/\)(([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?):/sprintf ")%.4f:", $1 * 100/eg' df9.tree
((((A_8:0.000846,(A_5:0.002449,(A_1:1e-06,((A_4:1e-06,((A_7:1e-06,A_6:0.001061)71.4000:1e-06,A_3:1e-06)31.4500:1e-06)35.8667:1e-06,A_2:1e-06)36.1000:1e-06)43.4800:1e-06)68.3500:0.001619)88.8571:0.001931,A_9:0.00069)68.8471:0.000691,...)

The following may also work, but there may be some numbers it won't match:

perl -pe 's/\)([-0-9.eE+]+):/sprintf ")%.4f:", $1 * 100/eg'
9
  • Hi thanks! Multiplication doesn't occur on the targeted number. Is it because of the extra : before the closing quote?
    – web
    Jul 8, 2021 at 12:32
  • 1
    yes, and it's precisely those numbers between ) and : that this perl code multiplies by 100. BTW, that docdiff command should have been docdiff --char --tty out.txt expected.txt (i.e. --tty not --lf, otherwise it gives html output).
    – cas
    Jul 8, 2021 at 12:38
  • 1
    the only way it can not produce the output you asked for is if there's something about the input format you haven't told us, or if your sample input is not a representative sample. e.g. is it possible that the input sometimes contains numbers in scientific notation (rather than plain decimal notation) between ) and :? If that's the case, try adding an e to the bracket expression in the regex: i.e. [-0-9.e]
    – cas
    Jul 8, 2021 at 12:48
  • 1
    Yeah, well, if you're using perl, you don't need to pre-process the input file with sed or awk. perl's like an over-powered superset of sed, tr, and awk combined (and that's even before you take into account the enormous library of CPAN modules). I updated my answer to use your df9.tree filename to make it clearer.
    – cas
    Jul 8, 2021 at 14:05
  • 1
    BTW, I'm not familiar with newick format files but it seems that it's a bioinformatics data format. You might want to take a look at BioPerl or bioinformatics.stackexchange.com. There's also a perl module on CPAN called Bio::PhyloNetwork for working with newick data. See also metacpan.org/search?q=newick to search for other perl modules which mention newick.
    – cas
    Jul 8, 2021 at 14:09
2

With perl's s///e, you can use the evaluation step to determine if the match was numeric, and substitute accordingly:

perl -MScalar::Util=looks_like_number -pe '
s{\)\K.*?(?=:)}{ looks_like_number($&) ? $&*100 : $& }ge' file

With GNU awk, using a regular expression for the record separator:

gawk 'prevRT==")" && RT==":" && $0+0 == $0 {$0 *= 100} {ORS = prevRT = RT}
1' RS='[):]' file

This tests the numeric-ness of a record with the comparison $0+0 == $0.

0

Using POSIX sed in conjunction with the desk calculator dc and bash shell we can get the result as shown. First we compute the form of the string to be synthesized then synthesize it by pushing it onto bash.

echo 'echo "'"$(sed -e '
  s#)\([^:)]\{1,\}\):#)$(echo "4k100 \1*1/p"|dc):#g' < file)"'"'|sh

Or, we can eval the variable which has been pre-filled with the command to generate the output.

var='echo "'$(sed -e '
  s#)\([^:)]\{1,\}\):#)$(echo "4k100 \1*1/p"|dc):#g' < inp)\"

eval "$var"

Using the 4-argument split function in GNU awk which keeps track of both the fields and the delimiters.

awk '{
  split($0, a, /[:)]/, s)
  for (i=1; i in a; i++)
    print (s[i-1] s[i] == "):" ? sprintf("%.4f",a[i]*100):a[i]) s[i]
  $0=RS
}1' ORS= file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.