I've inadvertently discovered a Proxmox bug. I run a personal cluster of 4 nodes (a technology lab for giggles and gags). Each node has a Dell Perc 5i and 4 HDDs in RAID 10. The standard Proxmox install uses the entire logical drive and puts in place 2 partitions, the system drive and a local drive (
I recently discovered that running QCOW2 allows me to take snapshots whereas running VMDK's does not. I also learned that I cannot migrate VMs unless they are on network storage.
Not a problem, first I install NFS server on all the nodes, then created and export a NFS folder on
/var/lib/vz file system (
/var/lib/vz/nfs) then added this as network storage.
Next I began to migrate my drives to the NFS folder on the local machine, and converting the images to QCOW2 using the web interface drive move command. This works, but here is how I can consistently crash the Proxmox cluster service:
- If I try to copy more than one VM drive at a time on the master node.
- If I try to copy a very large (250Gb) VM by itself on the master node.
The master node seems to just go south. The web interface shows everything is down. Progress on the copy just halts. OS sync commands run at the shell never return, shutdown requests hang, init 0 commands hang. Eventually, I have to cycle the power on the node.
So, some questions:
- Is it because I'm copying to NFS? I'm using a bog standard nfs4 nfs-kernel-server.
- Is it because I'm keeping the drive that hosts the master quorum too busy? After all, the RAID 10 setup is reading and writing these large files from local to a nfs share on the same machine - we are keeping the TCP stack and the drive itself very busy.
- Is there some best practice I'm missing here? - I know I would never put this into production - keep in mind this is a home-made cluster running a lab of inexpensive servers.
Any help would be appreciated.