I have a shared memory server with PBSpro installed as the job manager. There are 320 cores total, and pbs is configured so there is 1 job queue having 30 of the 32 cpu's so 300 physical cores to be divided up among users, with 2 cpu's or 20 cores left for head node operation.
some software will run on or parallelize over 100+ cores, and i'd like to verify N processes from user A are on unique and separate core numbers than the M processes from user B.
If I use
top I can do
hit F to get into sort menu hit J to sort by P = Last CPU used (SMP)
this gives me a P column in the top output having core number, but on a 300 core system I can only get around 70-80 rows before I run out of screen, on a 1920x1200 monitor. I can start shrinking font size in the terminal window but in the end I won't be able to see up to 300 rows, not that I can even visually process all that with top updating every 1..3 seconds.
My goal is to quickly and easily
- verify users are running stuff in the job queue within the correct core numbers, and not on the head node
- verify for a given cpu core that is at 100%, or anything over 50%, that only one process from one user is running on it. I want to make sure that if user A with programA.x is on core #234, user B with anything.x is NOT on core #234.
what's the best way to do this, when a single image shared memory server has many cores?