Why is my float comparison in awk not going the expected result?

Question

I have the following awk script:

{
    if ($1 > 1000) {
        print $0
    }
}

It should print all the lines where the value of the first and only column is above 1000.

Here is the test data:

1,151
1001,055
756,75788

Using awk -f my_script.awk my_data, I have the following output:

1001,055
756,75788

Where I am expecting:

1001,055

Awk version is:

GNU Awk 5.0.0, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.1.2)

What I am doing wrong ?

Edit:

as said in the comments:

Coma is not a separator here, it is the decimal separator, used in french and according to wikipedia, in all notation systems except the english one.

Edit 2: There is only one column in the example data. In the real data, field separator is ";".

Answer 1

Disclaimer The following, first solution is outdated due to a misinterperation of the original question. Please refer to edits 1 & 2 for matching solutions.

---

awk does not recognise commas as separators by default. It does so for tabs and spaces only. You will thus need to define the separator explicitly, otherwise awk compares a string value.

BEGIN {FS=","}
$1 > 1000

Note that I also used the simplified notation where a line is printed when a condition is met. This just as a hint for simpler code.

Or specify the separator on the command line:

awk -F,  -f script.awk infile

---

Edit 1 following specification that , is to be used as decimal separator. Please be aware that awk will consider . as decimal separator and using the locale settings for decimal separators is usually troublesome.

For Option 1, I'd suggest using a little trick: Still take integer and fraction as separate, comma-delimited fields and evaluate them individually:

 BEGIN {FS=","}
 $1==1000 && $2>0 || $1 > 1000

This will a) skip trying to use locale in awk and b) skip trying to translate back and forth between ,- and .-separation. The disadvantage being that in case there is more floating point data, the field numbers might not match column headers. However if it really is just about printing matching lines, this does not play a role.

An infile like

1,151
1001,055
756,75788
1000
1000,00
1000,000001

would return

1001,055
1000,000001

---

Edit 2 Another, probably more elegant, option is translating the first field into a dot-separated floating for the comparison:

gensub(/,/,".","g",$1)+0 > 1000

This works as follows: Interpret field 1 as string, replace , by ., add 0 to make it into a number in awk-logic, do the comparison and print if condition is true. The advantage is that with the specification of ; as field separator, this solution does not introduce problems with field numbers.

---

In general, I'd suggest trying to avoid , as decimal separator as far as possible. Of course this depends on who provides the data.

Answer 2

To put the comments from @Ed Morton and @steeldriver into an answer, you can have GNU awk treat the comma as the decimal separator if you make sure to use a locale that defines it so, and enable either --posix or --use-lc-numeric/-N.

E.g.:

$ LC_NUMERIC=fi_FI.UTF-8 awk -N '$1 > 1000' data.txt 
1001,055

or:

$ LC_NUMERIC=fi_FI.UTF-8 awk --posix '$1 > 1000' data.txt 
1001,055

As long as it considers only the dot as a decimal separator, stuff like 756,75788 doesn't get recognized as a number, but is instead treated as a string, and the comparison is string-based. 7 sorts after 1, and , sorts before 0, so 756,75788 > 1000 and 1,151 < 1000. (Though I'm not sure if it uses the locale's collating rules too, that might affect how the , is interpreted.)

You might try to force it to treat the value as a number by using ($1 + 0), but that would only look at the part before the comma, since in itself that wouldn't make it take the comma as a decimal separator. With the data in the question this might seem to work, but it would turn e.g. 1000,1 into 1000, which would not be printed. If you wanted to check for "at least 1000", instead of "greater than 1000", then you could use ($1 + 0) >= 1000 and just ignore the fractional part.

See: 6.1.4.2 Locales Can Influence Conversion and 6.3.2.1 String Type versus Numeric Type in GNU awk's manual. _{(The example on the latter page is silly, since 37 < 42 regardless of if the comparison is textual or numeric.)}