My view on this issue is from the developer side. I write the code that gets placed on a RHEL virtual machine running as one of many in an enterprise system. The filesystem being used is a remote, network attached storage device.

We had some high variability on simple commands during batch. So we set up a test to get more information, but now I don't know what we've found.

We ran the following command every 30 minutes and logged the output. It's a copy of a 6 gb file. What I see is elapsed time jump from 11 seconds to 190 seconds when the system is busy running lots of jobs and this test command gets low CPU time.

What I can see is that column "I" (filesystem inputs) gets populated when the CPU is low, but not when it's high. Column "w" (involuntary swaps) is also much higher.

My question is, what is happening to this job/command that forces it to run SO MUCH LONGER when the CPU time goes down? Does the swap in/out store all of that data on some other device that is much slower? Generally, what happens during a swap in/out?

Command being run:

/usr/bin/time -a -o filename.txt cp file.txt fileCopy.txt
Date Time e S U P c w I O
3/14/2022 5:19:02 64.9 16.23 1.03 26% 3005 29210 12000016 12000000
3/14/2022 5:49:02 12.7 11.63 0.79 97% 2069 76 0 12000000
3/14/2022 6:19:02 100.39 14.74 0.78 15% 1034 29925 12000136 12000000
3/14/2022 6:49:24 191.32 18.86 0.94 10% 3374 36164 12001024 12000000
3/14/2022 7:19:02 71.61 15.61 0.88 23% 1610 30316 12000296 12000000
3/14/2022 7:49:02 70.73 17.5 0.91 26% 1408 29540 12000072 12000000
3/14/2022 8:19:02 10.95 9.89 0.7 96% 1709 75 0 12000000
3/14/2022 8:49:02 11.01 10.22 0.73 99% 239 85 0 12000000

The column descriptions from the man page for /usr/bin/time

e   Elapsed real time (in seconds).
S   Total number of CPU-seconds that the process spent in kernel mode.
U   Total number of CPU-seconds that the process spent in user mode.
P   Percentage of the CPU that this job got, computed as (%U + %S) / %E.
c   Number of times the process was context-switched involuntarily (because the time slice expired).
w   Number of waits: times that the program was context-switched voluntarily, for instance while waiting for an I/O operation to complete.
I   Number of filesystem inputs by the process.
O   Number of filesystem outputs by the process.

1 Answer 1


This answer may not be specific to your example, but it is a more general answer to the question Generally, what happens during a swap in/out? Please note that some of this will be generalization, there is a lot of stuff I'm glossing over to avoid writing a doctoral thesis on memory management.

The short answer is that everything is about memory management. Both inside the container and outside the container. Let's first look at a very simple case - reading something from disk. First, the system will look at the variable you're reading the data into, and will request that much memory space from the kernel. Since we're in a newly-started system, the kernel easily allocates the memory, hands it to the process, and you copy the value from disk to memory. Simple, easy, quick.

So let's add some complexity. As the system starts more and more jobs, the CPU gets busier (obviously!). While this doesn't relate directly to memory management, it does relate to which processes get slices of the CPU's time. The simplest scheduler to understand is a round-robin scheduler - the kernel goes through it's list of processes asking for CPU cycles, and gives each one the same amount of cycles in the order they appear on the list. Which is most often the order in which they added themselves to the list. Then you add in the concept of process priority, which means certain special processes get to the top of the list quicker than others. Generally, the priority of this type of command (cp) will be fairly low, and it won't really ask for CPU time too often, because the majority of it's time will be spent waiting on disks to respond.

As an aside, the fact that you're reading from network storage adds more delay, since you're not only asking for the disk devices to give you data, but you're also asking the network interface to do stuff for you. Again, not directly related to the memory management discussion, but network disks are slower than local disks so you're spending even more time inside the process waiting on the I/O requests to complete.

Now, let's look at a system that has been running for a while. Most of the time, you'll see the "free" memory value be rather low, often unexpectedly so. Does this mean Linux is wasting your RAM? No, not at all. Part of the memory consumption is disk caching - see Linux At All My RAM! for a background on this aspect.

The other part of the high memory usage is that Linux is lazy - it doesn't do anything it doesn't absolutely have to. One of those things is returning memory pages to the "free" list. When a process finishes using a memory page, the kernel will mark that page as 'clean', but will leave it on the "used" list. This simply means that the page is available for immediate re-use by any other process. Note that a page can also be marked 'dirty' - this means that the process that was using this memory is finished with it and asked that it be written back to disk, but because no other process needs that particular page, Linux is waiting until someone else actually needs it to actually flush it to disk and mark the page as usable. This is most often seen in disk caching, but can also be seen in file writes.

So, we have a system that has nothing on the "free" list, but has a lot of pages on the "Clean" and "dirty" lists. Along comes a new process, asking for some memory, Linux then has to find a big enough chunk of space on either the "clean" or "dirty" lists - and if it's dirty, it has to force that page to be flushed to disk. In human terms, this happens just as instantaneously as allocating a page from the "free" list, but it computer terms it actually takes quite a bit longer.

In your case, while the container itself is new and inside the container all the memory is "free", the OS running the container likely has very little left on "free" and is instead allocating from the "clean" and "dirty" page lists. When the system is busy, the flushing of dirty pages happens more often, increasing IO wait times, increasing times to allocate memory, and so on.

Bottom line, your system is behaving normally. It's supposed to make unimportant processes like a cp take longer (increased system and user time slices from time) while it worries about all the other processes it's running, and especially about the higher priority processes it has. And yes, it really would be possible to write a doctoral dissertation on Linux memory management - it's that complex.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .