Why is Gnome fractional scaling 1.7518248558044434 instead of 1.75?

Question

If I set 175% scaling in Gnome Settings, the value is saved as 1.7518248558044434 in ~/.config/monitors.xml:

<monitors version="2">
  <configuration>
    <logicalmonitor>
      <x>0</x>
      <y>0</y>
      <scale>1.7518248558044434</scale>
      <primary>yes</primary>
      <monitor>
        <monitor spec>
          <connector>DP-3</connector>

Why is it so? At first, I thought it could be due to floating point rounding error, but 1.75 is one of those happy numbers whose the exact value can be expressed.

Gnome Wayland 43.3

Answer 1

The preset scale factors (100%, 125%, etc.) get adjusted to the closest values that give a whole number of pre-scaling virtual pixels both horizontally and vertically for your resolution; judging by your value of 1.7518248558044434 this is probably 2192 x 1233 and you have a 3840 x 2160 display.

Also as to why the width you would calculate with that value, 3840/1.7518248558044434 = 2191.9999520937613, is only accurate to about four places after the decimal point, clearly the scale has been converted from single-precision floating point (IEEE-754 32-bit). The double precision approximation of 3840/2192 is more like 1.7518248175182483, but if you convert that value to single-precision and back to double-precision you get 1.7518248558044434 precisely. I did it with Python, as suggested by the answer https://stackoverflow.com/a/43405711/60422:

>>> struct.unpack('f', struct.pack('f', 1.7518248175182483))[0]
1.7518248558044434

Stéphane Chazelas suggests the corresponding one-liner in Perl:

perl -e 'printf "%.17g\n", unpack "f", pack "f", 1.7518248175182483'

Why converting a floating point number to a higher precision gives a decimal representation with more digits that are of no use is the kind of floating point rounding error the question is alluding to -- the internal representation of the floating point number is in binary, and so the digits after the floating point internally (the "binary point" since it's binary) represent power of 2 fractions (1/2, 1/4, 1/8, and so on). A number you can express in a finite number of places in decimal does not necessarily have a finite representation in binary. For more on this see: https://stackoverflow.com/a/588014/60422

Single precision is generally said to be good for about 7 decimal significant figures and that's what we're seeing here.

To get an idea of how the adjustment of the scale factor that comes up with this number actually works, the get_closest_scale_factor_for_resolution function in mutter calculates the virtual width and height from the scale factor, and then if these aren't whole numbers, starting from the calculated width rounded down it tries whole number widths around the calculated one on both sides, expanding outward from it one pixel at a time, until it finds a width that gives an adjusted scale factor that would also make the virtual height a whole number, or until it gives up because the scale has gone out of range or out of the search threshold. https://gitlab.gnome.org/GNOME/mutter/-/blob/176418d0e7ac6a0418eea46669f33c8e3b03c4bd/src/backends/meta-monitor.c#L1960

If you want to know why the developers decided to do this, I don't have the answer for that, but my guess is backwards compatibility: developers are used to peoples' monitors having whole numbers of pixels, and so this is what the existing software out there is designed for.

Answer 2

Another theory: the rational which 1.7518248558044434 is an approximation of is not 2192/1233 but the simpler 240/137 = 1.7518248175182481... (To get a rational that's closer, you'd need numerator and denominator larger by a factor of 1390. And yes, the decimal representations of multiples of 1/137 have an 8-digit cycle.) So there are several possibilities for the height and width in pixels which would give this ratio, including 2160 x 1233.

But, you say, 240/137 is close but not that close. Another good approximation is 3673843/2097152. To get a rational that's closer, you'd need numerator and denominator larger by a factor of thousands. 1/2097152 is 2^{-21}. So that suggests that 240/137 was stored in a binary floating-point with enough room for 22 mantissa bits: one bit to the left and 21 bits to the right of the binary point. (These bit counts neglect any trailing 0s there might be.) Then converted to decimal with far more precision than there was accuracy in that binary representation.