Fault-tolerant iSCSI root fs for Linux

Question

I am in the process of moving some Linux servers onto a virtualized environment with their filesystems mounted from LVM volumes, which are in turn hosted on a remote NAS via iSCSI. I am able to start them up and they run perfectly with no issues.

However, the NAS server is Windows-based and, when Microsoft issues patches, it automatically applies them and reboots. When it reboots, all of the virtual servers' filesystems detect errors and go into read-only mode. I have attempted to remount them as read/write, but the kernel has the filesystem flagged as write-protected, so this fails. The only way I've been able to find to recover is to shut the virt down, fsck its LVM volume, and restart it.

The virts mount these LVM volumes with an fstab entry of the form:

/dev/xvda2 / ext3 noatime,nodiratime,errors=remount-ro 0 1

or

/dev/xvda2 / ext4 errors=remount-ro 0 1

The virtual host OS also has an LVM/iSCSI mount from the NAS server (in the same volume group, even) which continues working in read/write mode despite these interruptions. Its fstab entry is:

/dev/mapper/nas6-dom0   /mnt/nas6   ext4    _netdev     0   0

This leads me to suspect that removing errors=remount-ro from the guests' fstab entries would provide fault-tolerance, but I'm a bit uneasy about doing that - if an actual error develops in the filesystem, I would expect that allowing continued writes to the fs could make things much worse in short order.

What is the best practice for resolving this such that the virtual guests will be able to continue running after the NAS reboots itself?

Answer 1

According to the Open-iSCSI documentation:

8.2 iSCSI settings for iSCSI root

When accessing the root partition directly through a iSCSI disk, the iSCSI timers should be set so that iSCSI layer has several chances to try to re-establish a session and so that commands are not quickly requeued to the SCSI layer. Basically you want the opposite of when using dm-multipath.

For this setup, you can turn off iSCSI pings by setting:

node.conn[0].timeo.noop_out_interval = 0

node.conn[0].timeo.noop_out_timeout = 0

And you can turn the replacement_timer to a very long value:

node.session.timeo.replacement_timeout = 86400

The replacement_timeout setting defaults to 120 seconds and the NAS reboot took just over two and a half minutes, so this timeout was exceeded and the iSCSI session was discarded, along with all pending I/O requests, thus causing the virtual servers to see a disk failure and go to read-only mode.

Changing the timeout setting as described above should prevent this in the future, at least for NAS outages of up to 24 hours. And if it's down for longer than that, there are bigger problems to deal with anyhow.

Answer 2

There is also a way for change Recovery Timeout parameter on live, after connection is already established:

echo 86400 > /sys/class/iscsi_session/session28/recovery_tmo

replace session28 by your session id.