How to configure OpenBox to ignore the size hint of a specific application's window?

Question

Updated Question

Based on the excellent comment by @mosvy it is actually the window manager that is respecting or ignoring the propagated size hint of a window. Unfortunately though, I couldn't find a way to either not send size hint from urxvt or ignore the sent one by OpenBox.

So I guess my question really is: Is there a way at the urxvt level to stop sending size hint? Or is there a way to configure OpenBox to ignore the sent size hint for a specific application?

---

Original Question: 'urxvt gapless window size'

I'm using OpenBox as my window manager, and I have some hot keys configured to move windows into certain areas of the screen with a specified window size. All windows are respecting this, except urxvt which is calculating its size based on the font size of the characters it is displaying. However, if urxvt is in fullscreen mode^†, it is capable of filling the screen without gaps. (I assume it is just using the configured X background and filling in the remaining space aroung the window — but I can be wrong about this.)

So my question is this: is there a builtin configuration option or maybe a perl extension that makes urxvt fill all the space the window manager is giving it to without gaps when it is not in fullscreen mode?

---

^†Probably worth mentioning, that I'm using a perl extension already for a proper fullscreen experience.

Answer 1

The easiest way to fix it is to modify either the source of rxvt or the source of openbox and to recompile -- look at the bottom of this answer for two very simple patches.

Another way is use an LD_PRELOAD hack that will turn off the PResizeInc flag when getting the WM_NORMAL_HINTS window property, and will prevent it from being turned on when setting that property. This will only affect the width_inc and height_inc properties -- programs that set their minimum window size or its aspect ratio will continue to work fine.

$ cat no_inc_size_hints.c
#define _GNU_SOURCE
#include <dlfcn.h>
#include <X11/Xlib.h>
#include <X11/Xutil.h>

Status XGetWMNormalHints(Display *dpy, Window w, XSizeHints *hp, long *rp){
    static typeof (XGetWMNormalHints) *orig;
    Status r;
    if(!orig) orig = dlsym(RTLD_NEXT, "XGetWMNormalHints");
    if((r = orig(dpy, w, hp, rp)))
        hp->flags &= ~PResizeInc;
    return r;
}
void XSetWMNormalHints(Display *dpy, Window w, XSizeHints *hp){
    static typeof (XSetWMNormalHints) *orig;
    if(!orig) orig = dlsym(RTLD_NEXT, "XSetWMNormalHints");
    hp->flags &= ~PResizeInc;
    orig(dpy, w, hp);
}
$ cc -shared -fPIC -Wall no_inc_size_hints.c -ldl -o no_inc_size_hints.so

Then urxvt or openbox will have to be run with the

LD_PRELOAD="$LD_PRELOAD /absolute/path/to/no_inc_size_hints.so"

variable in their environment. Example:

$ LD_PRELOAD=`pwd`/no_inc_size_hints.so openbox --replace   

There are two problems with this:

1) urxvt is usually installed as a utmp setgid binary, and the LD_PRELOAD variable is cleared when exec'ing a set[gu]id binary. Look in the man page for why the utmp permission is (not) needed. So you'll have to copy the urxvt executable elsewhere (which will turn the setgid bit off). Example:

$ cp `which urxvt` .; LD_PRELOAD=`pwd`/no_inc_size_hints.so ./urxvt

2) The window manager is usually called through ssh-agent from the /etc/X11/Xsession.d scripts on many systems, and ssh-agent will also wipe the LD_PRELOAD variable from the environment. So something like this may be needed:

# cat <<'EOT' >/etc/X11/Xsession.d/98-no_inc_size_hints
export LD_PRELOAD="$LD_PRELOAD /path/to/no_inc_size_hints.so"
case $STARTUP in
/usr/bin/ssh-agent*)
        STARTUP="/usr/bin/ssh-agent env LD_PRELOAD=$LD_PRELOAD ${STARTUP#* }";;
esac
EOT

Possible patch for openbox-3.6.1:

--- openbox/client.c~   2018-10-06 08:34:25.615967414 +0300
+++ openbox/client.c    2018-10-06 08:34:28.916133702 +0300
@@ -1757,9 +1757,6 @@ void client_update_normal_hints(ObClient
         if (size.flags & PBaseSize)
             SIZE_SET(self->base_size, size.base_width, size.base_height);

-        if (size.flags & PResizeInc && size.width_inc && size.height_inc)
-            SIZE_SET(self->size_inc, size.width_inc, size.height_inc);
-
         ob_debug("Normal hints: min size (%d %d) max size (%d %d)",
                  self->min_size.width, self->min_size.height,
                  self->max_size.width, self->max_size.height);

and for rxvt-unicode-9.22:

--- src/main.C~ 2018-10-06 08:33:08.580085731 +0300
+++ src/main.C  2018-10-06 08:33:37.549545455 +0300
@@ -657,7 +657,7 @@ rxvt_term::window_calc (unsigned int new
   unsigned int w, h;
   unsigned int max_width, max_height;

-  szHint.flags = PMinSize | PResizeInc | PBaseSize | PWinGravity;
+  szHint.flags = PMinSize | PBaseSize | PWinGravity;
   szHint.win_gravity = NorthWestGravity;
   /* szHint.min_aspect.x = szHint.min_aspect.y = 1; */

@@ -1073,7 +1073,7 @@ rxvt_term::resize_all_windows (unsigned
     {
       szHint.flags &= ~(PBaseSize | PResizeInc);
       XSetWMNormalHints (dpy, parent, &szHint);
-      szHint.flags |= PBaseSize | PResizeInc;
+      szHint.flags |= PBaseSize;
     }

   if (!ignoreparent)

Both are against the versions from debian 9.5; they'll only apply with patch -l -- I don't know how to get this damn thing to preserve tabs.