So I'll describe the set-up, then the exact requirements and then the list of options I have tried and then I'll ask if their's a better approach or the best option among the ones mentioned.
So we are a group of Machine Learning researchers, We have one very powerful workstation machine, and other decently powerful machines one for each of us.
That the GPU is efficiently or equally allocated among all the active users at any given time while all users are working on the workstation simultaneously. (Ram is huge enough to not worry about and also we don't mind having common hard disks) (Some kind of GPU Virtualization?) We are looking for an approach that's up and running in 2-3 days.
The working OS is Ubuntu 16 on all the machines
The Proposals :
- Setting up multiple VMs in the Workstation, one per user and SSH from our current machines. Running a VM over another OS seems like a big overhead plus we'd rather like to spend on more hardware than software licenses. VMWare ESXI bare-metal seems one way to go.
- The multiseat approach, it would allow multiple users at the same time, though it requires one set of keyboard, mouse and video card per seat, we do have a very powerful GPU dedicated just to the display but again it's just one and multi-seat requires one per seat, while there are slow workarounds to operate with a single video card(xephyr) we'd still need to allocate the computing GPU among users efficiently.
- Multiple users SSH into multiple Virtual Terminals. The multiple Virtual Terminals in Unix were made in the time where the computers were expensive and a single computer would be shared among different users using Terminals. We'd still need a way to virtualize the GPU. But if all else works good we can still work since their are four users and two computing GPUs so we could run two programs at once assinging each to one GPU manually through the code(Tensorflow), but if there's an approach to virtualize the two physical GPUs into 4 virtual GPUs it'd be best(except Nvidia vGPU).
- rCUDA, have sent them a request form. Waiting.
- Some cluster management system such as Apache Mesos. Since single or multiple computers a CMS won't mind and it's made to virtualize and allocate it's resources efficiently among it's clients.
- LTSP, haven't looked much into it.
Now I know I might sound naive in many of above suggestions, so please give a suggestion as per your knowledge. In case anything in the question seems vague please point to it and I'd clear it out.