3

According to section 3.2.9 on page 39 on コンパチブル・シェルスクリプティング(第5版), a reference book of how to make a compatible shell script, there are some AWK implementation with limited range of integer (translation by me):

For example, wouldn't your AWK display such the following output?

$ awk 'BEGIN{print 2147483648}'
2.14748e+09
$

That is an implementation who cannot handle integers more than 0x7FFFFFFF (maximum value of 4-byte signed integer). This is why you should be careful with handling integers with many digits. If you just want to display them without calculation, you should treat it as a string.

But I have never found out what EXACTLY the portable range was, on the book. So I referred POSIX documents to find out that SUSv2 seems not to specify the range, and it simply says:

If the value is too large or too small to be representable, the behaviour is undefined.

On 2004 edition, it seems that integer and float values are signed long-typed and double-typed, respectively ('...' indicates that I omitted the part):

Integer variables and constants ... shall be implemented as equivalent to the ISO C standard signed long data type; floating point shall be implemented as equivalent to the ISO C standard double type.

Does that mean that [-2147483647,+2147483647] (PS. I looked up the range just on Wikipedia) is the portable range of integer that would not be handled as float?

8
  • 2
    To me, that's what it means. The most common implementation used in Linux distros, GNU awk, could be considered a de-facto standard and can use arbitrary precision integers. Mar 20, 2021 at 3:51
  • 2
    Note that a 32 bit signed long would allow -2147483648 .. +2147483647, but mawk (at least mawk 1.3.4 20200120 on Debian amd64 here), still prints -2147483648 as float even though longs are 64 bit on that system. Also note that if you want to consider the /bin/awk of Solaris, even awk 'BEGIN{print 0}' is not portable, as that's the original awk API from the 70s there, where you'd need awk 'BEGIN{print 0; exit}' for awk to exit after printing that 0. Mar 20, 2021 at 6:46
  • FWIW, with the original implementation in Unix V7 from the late 70s, tested on a pdp11 emulator, the range is -2147483648 .. 2147483647 (it prints -2147480000 for -2147483649 and 2147480000 for 2147483648). Mar 20, 2021 at 7:01
  • @Stéphane, thank you for researching a lot; so we can presume that [-2147483647,+2147483647] IS the portable range (including histotical environments) (also modern ones have wider range), can't we? Also thanks for correcting my post.
    – user451199
    Mar 23, 2021 at 5:21
  • 1
    If the number is being truncated from 2147483647 to 2147480000 the value is not preserved. The correct range is another @StéphaneChazelas
    – user232326
    Mar 24, 2021 at 2:31

4 Answers 4

1

I wonder what exactly you mean with the number being handled as an integer, and not a float.

If you mean what printf "%d" will output, then it appears -2147483647 is safe in gawk, mawk and Busybox. Numbers lower than that print as -2147483647 in mawk, and -2147483648 in Busybox, but gawk and whatever the awk on my Mac is, print the actual values.

On the other hand, if you mean doing calculations with the numbers, then you might be able to get a greater range. Awk should use whatever the "ISO C standard double type" is on the platform. The most common one is IEEE 754 double-precision floating point numbers, but that's not required.

With an IEEE double, the mantissa is 52+1 bits, so any integers within about ± 253 should be possible to represent exactly. Printing the numbers is just a question of the output format.

The default output format for print (OFMT) is %.6g, which implies 6 significant digits are printed. Except that it's not the whole truth, since integers are supposed to be printed as integers, but it depends on the version of awk what they count as integers here. Some limit it to numbers in a particular range, e.g.:

$ busybox awk 'BEGIN { a = 9007199254740992; print a; printf OFMT "\n", a }'
9007199254740992
9.0072e+15

vw.

$ mawk 'BEGIN { a = 9007199254740992; 
                print a; printf OFMT "\n", a }'
9.0072e+15
9.0072e+15

In any case, you should be able to change OFMT to e.g. %.0f to get mawk to print the full number too:

$ mawk 'BEGIN { OFMT="%.0f"; a = 9007199254740992;
                print a; printf OFMT "\n", a }'
9007199254740992
9007199254740992

Greater than ± 253, and you have problems because the lowest bits start dropping off:

$ awk 'BEGIN { OFMT="%.0f"; a=9007199254740990; 
               for (i = 0; i < 6; i++) print a, "+", i, "=", a + i; }'
9007199254740990 + 0 = 9007199254740990
9007199254740990 + 1 = 9007199254740991
9007199254740990 + 2 = 9007199254740992
9007199254740990 + 3 = 9007199254740992
9007199254740990 + 4 = 9007199254740994
9007199254740990 + 5 = 9007199254740996

Of course, the calculations are still done using floats, regardless of OFMT, so here you get 2000000 = 3 * 666666.666666, not 1999998 = 3 * 666666 unless you truncate to an int:

$ awk 'BEGIN{a = 2000000; b = a/3; print 3*b}'
2000000
$ awk 'BEGIN{a = 2000000; b = int(a/3); print 3*b}'
1999998

You probably should make a test script to verify the behaviour you need on the awk(s) you use.

5
  • 2
    While POSIX says or at least suggests awk should use the C compiler's double type for internal representation of floating point numbers, there's nothing that says they have to be implemented as IEEE 754 double precision numbers AFAICT. It also says integers have to be represented as long, but mawk doesn't. There aren't many awk implementations that are POSIX compliant out there. See my comment on the question about the V7 awk on PDP11, where 2147483648 exceeds the float precision limit. Mar 23, 2021 at 9:12
  • @StéphaneChazelas, yep, it's punted to the "Concepts Derived from the ISO C Standard" page, where it says "Integer variables and constants, ... used by the standard utilities listed in this volume of POSIX.1-2017 shall be implemented as equivalent to the ISO C standard signed long data type; floating point shall be implemented as equivalent to the ISO C standard double type.".
    – ilkkachu
    Mar 23, 2021 at 11:45
  • 1
    But ISO C doesn't say double has to be IEEE 754 double precisions AFAIK. Mar 23, 2021 at 11:56
  • Though the awk page says "All arithmetic shall follow the semantics of floating-point arithmetic as specified by the ISO C standard". I guess the note about converting exact integers to strings also applies to printing, even though the context only talks about CONVFMT, and not OFMT. E.g. this doesn't use OFMT for the first one: awk -v OFMT="%.3f" 'BEGIN{print 123, 12.3}'. But I guess there's differences in what they consider "exactly equal to integers". I get gawk 'BEGIN{a=1000000000000000000000000; print a}' as 999999999999999983222784. It's an integer, yes, but...
    – ilkkachu
    Mar 23, 2021 at 11:57
  • 1
    @StéphaneChazelas, and no, it doesn't have to be 754, you're right. I did mean to note something about division, but of course that doesn't stay in the integers regardless of the range. But granted, it's worse with OFMT masking it
    – ilkkachu
    Mar 23, 2021 at 11:58
0

just wanna quickly dispel a rather common misunderstanding of mawk : it's fully capable of IEEE 754 double-precision floating point, like any other awk - the only thing to be mindful of is to use %.f rather than %d %i %u for any integer more than 9 digits long :

 jot -s $'\n  ' -w '%2d' - 1 33 2 | 

 mawk 'BEGIN {
    printf("\n  ")
     _ += __=_^=FS="^$"
    ___ = __-(++_)^-(_^_+_+_)

    OFS = "-st/nd/rd/th-power-of-3 :: "
   OFMT = CONVFMT = "<( %\047"(_^_)".f )>" 

 } $++NF = _^$__ * ___^(+$__<_^_)'  
   1-st/nd/rd/th-power-of-3 :: <(                           3 )>
   3-st/nd/rd/th-power-of-3 :: <(                          27 )>
   5-st/nd/rd/th-power-of-3 :: <(                         243 )>
   7-st/nd/rd/th-power-of-3 :: <(                       2,187 )>
   9-st/nd/rd/th-power-of-3 :: <(                      19,683 )>
  11-st/nd/rd/th-power-of-3 :: <(                     177,147 )>
  13-st/nd/rd/th-power-of-3 :: <(                   1,594,323 )>
  15-st/nd/rd/th-power-of-3 :: <(                  14,348,907 )>
  17-st/nd/rd/th-power-of-3 :: <(                 129,140,163 )>
  19-st/nd/rd/th-power-of-3 :: <(               1,162,261,467 )>
  21-st/nd/rd/th-power-of-3 :: <(              10,460,353,203 )>
  23-st/nd/rd/th-power-of-3 :: <(              94,143,178,827 )>
  25-st/nd/rd/th-power-of-3 :: <(             847,288,609,443 )>
  27-st/nd/rd/th-power-of-3 :: <(           7,625,597,484,987 )>
  29-st/nd/rd/th-power-of-3 :: <(          68,630,377,364,883 )>
  31-st/nd/rd/th-power-of-3 :: <(         617,673,396,283,947 )>
  33-st/nd/rd/th-power-of-3 :: <(       5,559,060,566,555,523 )>

The last number, 3^33, is between 52- and 53-bits (~52.304 bits), and is the largest power of 3 supported by 754 double fp to full precision.

not only it has no bespoke concerns regarding integer ranges other than the same 2^53 - 1 limitation that all double-precision floating point software encounter,

it's also capable of directly adding fancy formatting syntax into CONVFMT/OFMT, thus allowing for direct output generation without an extra call to sprintf()/printf() for each row.

And this is the standard mawk 1.3.4 that's being widely distributed, instead of a custom compile :

mawk -Wv
mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs:       unknown
regex-funcs:        internal
compiled limits:
sprintf buffer      8192
maximum-integer     2147483647
9
  • Mawk is not like any other awk. All awk I could test handle integers up to 2^53, mawk changes at 2^31. And GNU awk -M handles all integers (within reason) as integers (yes GMP integers, but integers anyway). Aug 25, 2022 at 22:53
  • @QuartzCristal : set CONVFMT="%.250g"; OFMT="%.25g"then tell me again mawk can't handle integers up to 9E15 Aug 26, 2022 at 6:49
  • @QuartzCristal : the only thing changed is the default formatting of it (OFMT). There's nothing different in internal processing. You're absolutely imagining things. see my detailed response below Aug 26, 2022 at 15:24
  • @QuartzCristal : look at my full reponse below. i never once modified CONVFMT Aug 26, 2022 at 15:34
  • @QuartzCristal : gawk -Mbe 'BEGIN { print 1E25 }' 10000000000000000905969664 :::: so much for GMP handling big-integers well Aug 26, 2022 at 15:36
0

In fact, there are three specific limits in AWK.

AWK Integers

If by integer they meant this description:

The internal representation of value in AWK is always a float (usually a double float). When a value is used to generate an string, if that value is an integer, a format of %d is used (no matter what the CONVFMT is. Ergo: an integer will remain an integer when printed as text.

As extracted from the mawk manual:

A numeric expression is converted to string by replacing expr with sprintf(CONVFMT, expr), unless expr can be represented on the host machine as an exact integer then it is converted to sprintf("%d", expr).

In practice, that could be revealed with this:

$ mawk 'BEGIN{ CONVFMT="used" ; a=12 ; b = 2^35; c = a "" ; d = b ""; print c, d }'
12 used

Or, In a more usual way:

$ mawk 'BEGIN{ CONVFMT="%2.2f" ; a=2^12 ; b = 2^35; c = a "" ; d = b ""; print c, d }'
4096 34359738368.00

$ mawk 'BEGIN{ CONVFMT="%2.2f" ; a=2^31 ; print a-1"",a"",a+1"" }'
2147483647 2147483648.00 2147483649.00

As seen above, the value of 2^31-1 is printed an an integer, the other two as floats.

GNU awk and busybox awk don't show that problem (in a 64 bit Debian at least).

Float

Floats are limited by the number of binary digits that could be used. This is not a problem of how they are formatted but of what they are.

In mawk anything bigger than 2^31 will be printed as a float (with CONVFMT="%.2f").

$ mawk 'BEGIN{ CONVFMT="%.2f"; val=2^53; print val-1"",val"",val+1""}'
9007199254740991.00 9007199254740992.00 9007199254740992.00

But it also turns out that the biggest integer (without exponent) is 2^53. That is also true in GNU awk, but GNU awk doesn't use the %.2f format. What reveals the limit is that adding one to 9007199254740992 repeats the value again. The value is truncated to 53 bits.

$ awk 'BEGIN{ CONVFMT="%.2f"; val=2^53; print val-1"",val"",val+1""}'
9007199254740991 9007199254740992 9007199254740992

GMP library

When GNU awk is compiled with both the GMP and the FMPR libraries (which nowadays is the standard) all (within reason) integers are represented as integers:

$ awk -M 'BEGIN{ print 2^300; print 2^300+1}'

2037035976334486086268445688409378161051468393665936250636140449354381299763336706183397376
2037035976334486086268445688409378161051468393665936250636140449354381299763336706183397377

In this case, the limit is quite big (I didn't search for an specific memory limit, but be sure that one exists. Infinite numbers are not representable).

-3
mawk 'BEGIN { OFS="\t"; OFMT="%\44725.f"

    ____=___=_*=((_+=_^=_<_)^++_)^(_*_++)*(_______=_)
   ____*=(_=_______)^(_+_+_)
   _________=(_______*=_*_)^(++_+—_);—____

   for(______+=______=(_+=(_^=_<_)+_)^(_+_+_+_);_<______;_++) {
       for(__+=__^=_<_;__<_______;__++) {
           if( (________=_^__)<____ &&  ___<________ ) { 
               print "",_,__,________                } } } }' | 

mawk '+(/\t2\t/)<+(/[17]$/)' FS='^$' | mawk '!(NR % 1777)' FS='^$'
2719    4          54,655,872,347,521
5681    3             183,347,236,241
8641    4       5,575,143,118,268,161
15323   3           3,597,753,503,267
24211   3          14,191,822,905,931
33093   3          36,241,688,055,357
41981   3          73,987,497,479,141
50863   3         131,584,858,085,647
59751   3         213,321,944,741,751
68633   3         323,294,970,192,137
77521   3         465,862,871,291,761
86403   3         645,039,730,972,827
95291   3         865,277,983,727,171
104173  3       1,130,486,847,025,717
113061  3       1,445,234,988,645,981
121943  3       1,813,304,024,948,807
130831  3       2,239,401,592,646,191
139713  3       2,727,158,971,340,097
148601  3       3,281,445,502,325,801
157483  3       3,905,719,392,797,587
166371  3       4,605,034,424,282,811
175253  3       5,382,652,995,919,277
184141  3       6,243,836,065,115,221
193023  3       7,191,627,487,303,167
201911  3       8,231,518,131,421,031

you can replicate the code-run yourself.

literally the only things i set that weren't the defaults was using \t tab as output delimiter, and a comma-aligned output formatting.

not even once have i called sprintf()/printf() anywhere in the code, or access an external utility.

And i have the same mawk-1 as everyone else does :

mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs:       unknown
regex-funcs:        internal
compiled limits:
sprintf buffer      8192
maximum-integer     2147483647
2
  • @QuartzCristal : so ? that's purely a formatting thing. there's no precision loss inside the number itself. you can do all the processing and only change OFMT right before u print, and all ur precision is right there. you're thoroughly confused between a formatting truncation (what mawk is) and precision truncation (what u imagine it to be ) Aug 26, 2022 at 15:44
  • @QuartzCristal : no, you were arguing it had to change internal processing methods just to handle those numbers. Aug 26, 2022 at 15:57

You must log in to answer this question.