Why does head; tail on a large file sometimes take a long time and sometimes not?

Question

I am writing a bash script that reads the first 10 lines and the last 10 lines of a .txt file. It looks for started (head) and completed (tail) and compares the number of occurrences using grep. The files are quite large which is why I opted to only read the head and tail of the files instead of the entire text. However, when I run the script the large files take a long time to "finish up" (which consists of reading the first 10 lines and last 10 lines and the compare, a task that should only take a moment or two).

While watching the script output text, I noticed this issue. So I decided to see if it would take a similar amount of time when I simply executed the head/tail (plus grep, as to simulate what is executing in the script) command straight from the command line. Surprisingly, the commands executed near instantly. I thought this was strange and I ran the script again. This time, the script would scream through the file it was stuck on before until it got to the next "large" file I had not already run the head/tail/grep command on.

This got me thinking, does bash store the results of a command similar to caching? Also, what could be causing these commands:

 head -n 10 /file/path/myfile.txt | grep -w -c 'lead word'
 tail -n 10 /file/path/myfile.txt | grep -w -c 'end word'

to be taking so long to execute?

edit: The reason I believe it is the head/tail lines above that are the source of problem is because there are echo lines that should print upon completion of head/tail individually. I have checked the line size of the files and they are not terribly longer than ones that are finishing within a few moments.

Could someone give me a more detailed explanation of how head/tail works at a more technical level? I have a very basic understanding of "the first/last x lines" of a file.

Answer 1

No, bash does not cache the output of commands, since the output can change from run to run; bash has no sure-fire way to track whether the file has been modified by another process, which would be critical, so that bash could tell whether its cached result was still valid or not.

However, there are other things at play here. When you use a shell (such as bash), you're simultaneously interacting with multiple parts of the system. For example:

• The shell itself
• GNU Readline, which is a line-editing interface used by multiple shells and other tools
• The libc implementation, which can cause confusing behavior in nearly any program from time to time.
• The terminal itself, which may have quirky behavior, and can actually respond to commands of its own. (E.g. either of Backspack and Delete may be unavailable, or they may be interchanged)
• The GUI window the terminal exists inside of (if applicable). E.g. it may have an IME which allows typing Unicode characters into the terminal with a special key sequence (such as, Ctrl+Shift+u followed by a number.)
• The kernel, including all of its modules and drivers
• The hardware itself, which, of course, can overheat, short-circuit, lose power, etc

In this case, I'd say the biggest contributor is not bash itself, but the filesystem-level caching mechanism implement in the kernel. Once you've read a file once, a large chunk of it can end up in the filesystem cache, which is a hunk of memory reserved for this purpose.

When performing operations on the file a second time, the shell does not trigger a read of physical hardware again, but instead retrieves the file contents from the cache. Nearly any operation you (re)perform in bash will be extremely fast in comparison to a disk read. That's why you don't notice that bash is actually re-performing the commands, because the slow disk read is missing.