Proxmox Master Node crashes

Question

I've inadvertently discovered a Proxmox bug. I run a personal cluster of 4 nodes (a technology lab for giggles and gags). Each node has a Dell Perc 5i and 4 HDDs in RAID 10. The standard Proxmox install uses the entire logical drive and puts in place 2 partitions, the system drive and a local drive (/var/lib/vz).

I recently discovered that running QCOW2 allows me to take snapshots whereas running VMDK's does not. I also learned that I cannot migrate VMs unless they are on network storage.

Not a problem, first I install NFS server on all the nodes, then created and export a NFS folder on /var/lib/vz file system (/var/lib/vz/nfs) then added this as network storage.

Next I began to migrate my drives to the NFS folder on the local machine, and converting the images to QCOW2 using the web interface drive move command. This works, but here is how I can consistently crash the Proxmox cluster service:

1. If I try to copy more than one VM drive at a time on the master node.
2. If I try to copy a very large (250Gb) VM by itself on the master node.

The master node seems to just go south. The web interface shows everything is down. Progress on the copy just halts. OS sync commands run at the shell never return, shutdown requests hang, init 0 commands hang. Eventually, I have to cycle the power on the node.

So, some questions:

1. Is it because I'm copying to NFS? I'm using a bog standard nfs4 nfs-kernel-server.
2. Is it because I'm keeping the drive that hosts the master quorum too busy? After all, the RAID 10 setup is reading and writing these large files from local to a nfs share on the same machine - we are keeping the TCP stack and the drive itself very busy.
3. Is there some best practice I'm missing here? - I know I would never put this into production - keep in mind this is a home-made cluster running a lab of inexpensive servers.

Any help would be appreciated.

Answer 1

This sounds like you're running into a networking issue with Proxmox. This seems to be a fairly common problem with both Proxmox and OpenVZ. I use OpenVZ and have had similar issues with OpenVZ.

• snapshot backup to NFS crashes cluster

The recommended way to work around this issue is to split your networking up so that the storage is on it's own network segment separate from the network that the VM's are using. You'll often times see these messages along with this issue, ipcc_send_rec failed: Resource temporarily unavailable.

In my experiences with OpenVZ (and indirectly Proxmox) I would say these technologies are fine for development and for low end use but really aren't designed for heavy use. I realize you're not using it in a heavy utilization application but their inability to be "fault tolerant" is one of the main reasons I've been looking to move away from them (OpenVZ) to alternative technologies such as LXC and Docker. Just my $0.02.