TL;DR What is the best/simplest way to reboot a gluster node (containing a replicated brick) in such a way that it doesn't impact (i.e. freeze) the remaining cluster?
I'm configuring gluster in an N-way replicated setup, where each node contains a mirrored brick with all shared data, on up-to-date Debian 9.4 systems running the packaged version of glusterfs 3.8.8.
While testing what happens on reboot of a node, I saw the entire cluster blocking read/write access for 42 seconds, effectively freezing anything touching the share for that period. This was easily reduced to a few seconds by setting
network.ping-timeout to a low value, but still we keep the noticeable freeze.
Killing the glusterfsd running the brick in question, right before reboot, prevents the freeze. Log files show that glusterfsd is in fact killed during reboot, but only after networking is down, so it can't notify other bricks, that have to find out it's gone using the ping timeout.
Is handcrafting a service file to do a
killall (as is suggested at https://bugzilla.redhat.com/show_bug.cgi?id=1022542#c4) really the way to go here? I hope I'm missing something.