In our research group we are running a computing server for deep learning with a number of NVIDIA Titan X graphics cards and quite some CPU cores. Given that it is a research lab and we have ~10 people using the machine the load on the CPU/GPU cores is almost always high.
I am now in charge showing that the machine is overly used and I can propose hardware upgrades. To make an argument, I want to create a detailed history of the CPU/GPU/MEM usage on the machine. The problem is, I don't know the right tools for the job. Of course, I can do some scripting but I prefer off-the-shelf tools since I am not a system administrator :) For monitoring the CPU/GPU usage I typically use
htop, but these are not suitable for generating long-time histories.
Any recommendations on creating such histories?