1

I am in the process of moving some Linux servers onto a virtualized environment with their filesystems mounted from LVM volumes, which are in turn hosted on a remote NAS via iSCSI. I am able to start them up and they run perfectly with no issues.

However, the NAS server is Windows-based and, when Microsoft issues patches, it automatically applies them and reboots. When it reboots, all of the virtual servers' filesystems detect errors and go into read-only mode. I have attempted to remount them as read/write, but the kernel has the filesystem flagged as write-protected, so this fails. The only way I've been able to find to recover is to shut the virt down, fsck its LVM volume, and restart it.

The virts mount these LVM volumes with an fstab entry of the form:

/dev/xvda2 / ext3 noatime,nodiratime,errors=remount-ro 0 1

or

/dev/xvda2 / ext4 errors=remount-ro 0 1

The virtual host OS also has an LVM/iSCSI mount from the NAS server (in the same volume group, even) which continues working in read/write mode despite these interruptions. Its fstab entry is:

/dev/mapper/nas6-dom0   /mnt/nas6   ext4    _netdev     0   0

This leads me to suspect that removing errors=remount-ro from the guests' fstab entries would provide fault-tolerance, but I'm a bit uneasy about doing that - if an actual error develops in the filesystem, I would expect that allowing continued writes to the fs could make things much worse in short order.

What is the best practice for resolving this such that the virtual guests will be able to continue running after the NAS reboots itself?

|improve this question
  • The best practice is to secure your storage. You're trying to fix the wrong thing here, it's on the NAS side you need fault tolerance. – Mat Apr 21 '15 at 9:15
  • Fair point, but I don't control the NAS, so the only thing I can do about that side is to complain to the guy who is responsible. I don't expect much to happen in response to that, so it still leaves me in need of a way to keep my servers working after a NAS reboot. – Dave Sherohman Apr 21 '15 at 9:31
0

According to the Open-iSCSI documentation:

8.2 iSCSI settings for iSCSI root

When accessing the root partition directly through a iSCSI disk, the iSCSI timers should be set so that iSCSI layer has several chances to try to re-establish a session and so that commands are not quickly requeued to the SCSI layer. Basically you want the opposite of when using dm-multipath.

For this setup, you can turn off iSCSI pings by setting:

node.conn[0].timeo.noop_out_interval = 0

node.conn[0].timeo.noop_out_timeout = 0

And you can turn the replacement_timer to a very long value:

node.session.timeo.replacement_timeout = 86400

The replacement_timeout setting defaults to 120 seconds and the NAS reboot took just over two and a half minutes, so this timeout was exceeded and the iSCSI session was discarded, along with all pending I/O requests, thus causing the virtual servers to see a disk failure and go to read-only mode.

Changing the timeout setting as described above should prevent this in the future, at least for NAS outages of up to 24 hours. And if it's down for longer than that, there are bigger problems to deal with anyhow.

|improve this answer
0

There is also a way for change Recovery Timeout parameter on live, after connection is already established:

echo 86400 > /sys/class/iscsi_session/session28/recovery_tmo

replace session28 by your session id.

|improve this answer

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.