A process's resident set size is the amount of memory that belongs to it and is currently present (resident) in RAM (real RAM, not swapped or otherwise not-resident).
For instance, if a process allocates a chunk of memory (say 100Mb) and uses it actively (reads/writes to it), its resident set size will be about 100Mb (plus overhead, the code segment, etc.). If after the process then stops using (but doesn't release) that memory for a while, the OS could opt to swap chunks of that memory to swap, to make room for other processes (or cache). The resident set size would then decrease by the amount the kernel swapped out. If the process wakes up and starts re-using that memory, the kernel would re-load the data from swap, and the resident set size would go up again.
The ru_maxrss field of struct rusage is the "high water mark" for the resident set size. It indicates the peak RAM use for this process (when using RUSAGE_SELF).
You can limit a process's resident set size to avoid having a single application "eat up" all the RAM on your system and forcing other applications to swap (or fail entirely with out-of-memory conditions).
ru_maxrssis in bytes on mac / BSD machines.ru_maxssis in bytes on Mac OS X (Darwin) machines, but in kilobytes on BSD machines and Linux machines. See stackoverflow.com/q/40773924/781723 and stackoverflow.com/q/12050913/781723.