0

I have a scenario

where i am calculating the sum of particular column using below two command

can any one explain me in detail what actually the command is performing

First command [when used to calculate the sum of particular column ]

awk -F '"?\\|"?' '{T+=$(2)*1000} END {printf "%.2f\n",T/1000}' demofile.txt

Second command [when used to calculate the sum of particular column ]

awk -F '"?\\|"?' '{T+=$(2)} END {printf "%.2f\n",T}' demofile.txt

when sum calculated using both command is different. why is it so ?

This is the Output : enter image description here This is the file used for calculating [please download and test] (link removed by moderator, possibly security concern)

14
  • 1
    Is it different? Not for me. Show input file and (different) results.
    – RudiC
    Commented Apr 14, 2020 at 9:32
  • @RudiC i will edit the question with demo example Commented Apr 14, 2020 at 9:36
  • 1
    @KamilMaciorowski its a great question can look into this issue i will share the demo example give me time i am editing the question Commented Apr 14, 2020 at 9:37
  • 1
    @kawenal6962go-mail.com Please add a representative sample of the data to the question itself. Data kept on a remote server may be gone tomorrow, which renders your question useless. Since there was security concern about the link and the site in your follow-up question, I have removed the link from this question.
    – Kusalananda
    Commented Apr 15, 2020 at 8:47
  • 1
    @Kusalananda, there is no "representative sample" to that problem. You need a huge file, because the error for each sum is very little. But to demonstrate the issue, you can just use yes 0.1 | head -n 10000000. E.g. yes 0.1 | head -n 10000000 | awk '{sum+=$1}END{printf "%.5f",sum}'
    – pLumo
    Commented Apr 15, 2020 at 8:53

1 Answer 1

4

The difference is because of what the gawk manual states:

Binary floating-point representations and arithmetic are inexact. Simple values like 0.1 cannot be precisely represented using binary floating-point numbers, and the limited precision of floating-point numbers means that slight changes in the order of operations or the precision of intermediate storage can change the result. To make matters worse, with arbitrary-precision floating-point arithmetic, you can set the precision before starting a computation, but then you cannot be sure of the number of significant decimal places in the final result.

gawk is GNU awk. It supports -M:

-M
--bignum

Select arbitrary-precision arithmetic on numbers. This option has no effect if gawk is not compiled to use the GNU MPFR and MP libraries.

Your awk may or may not be equivalent to gawk. In my Debian 9 each of the following two commands yields 25396577843.76:

LC_NUMERIC=C gawk -M -v PREC=60 -F '"?\\|"?' '{T+=$(2)*1000} END {printf "%.2f\n",T/1000}' demofile.txt
LC_NUMERIC=C gawk -M -v PREC=60 -F '"?\\|"?' '{T+=$(2)} END {printf "%.2f\n",T}' demofile.txt

Although with printf "%.4f\n",T I can still see the difference. Increase PREC to get better results.

The underlying problem is explained on this site:
What Every Programmer Should Know About Floating-Point Arithmetic

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .