I am running a large simulation on a computer cluster using 50 compute nodes. This solver uses a data structure which grows on the fly and (very) differently for each node. I need to make sure the memory used does not grow beyond each node's memory limit.
So far, I am doing it in the most inefficient way: I have one terminal tab open for each node and run
top to check the % memory used.
Is there a way I can do it with a script? The idea would be to ssh on each node and store the memory usage, ssh to the next, etc...