2

I am writing a bash script that reads the first 10 lines and the last 10 lines of a .txt file. It looks for started (head) and completed (tail) and compares the number of occurrences using grep. The files are quite large which is why I opted to only read the head and tail of the files instead of the entire text. However, when I run the script the large files take a long time to "finish up" (which consists of reading the first 10 lines and last 10 lines and the compare, a task that should only take a moment or two).

While watching the script output text, I noticed this issue. So I decided to see if it would take a similar amount of time when I simply executed the head/tail (plus grep, as to simulate what is executing in the script) command straight from the command line. Surprisingly, the commands executed near instantly. I thought this was strange and I ran the script again. This time, the script would scream through the file it was stuck on before until it got to the next "large" file I had not already run the head/tail/grep command on.

This got me thinking, does bash store the results of a command similar to caching? Also, what could be causing these commands:

 head -n 10 /file/path/myfile.txt | grep -w -c 'lead word'
 tail -n 10 /file/path/myfile.txt | grep -w -c 'end word'

to be taking so long to execute?

edit: The reason I believe it is the head/tail lines above that are the source of problem is because there are echo lines that should print upon completion of head/tail individually. I have checked the line size of the files and they are not terribly longer than ones that are finishing within a few moments.

Could someone give me a more detailed explanation of how head/tail works at a more technical level? I have a very basic understanding of "the first/last x lines" of a file.

  • Probably not. However, the filesystem driver in the kernel certainly has a cache. – jpaugh Aug 17 '17 at 15:10
  • 1
    It could take long if those 10 lines were worth several hundred megabytes. Or if /file or /file/path were very big non-indexed directories. – Stéphane Chazelas Aug 17 '17 at 15:11
  • 1
    @jpaugh Stéphane talks about a huge line size, not a huge file size. Which is not the case according to the OP. – xhienne Aug 17 '17 at 15:16
  • 1
    I think the issue lies in your script. Do you make large writes in your script? This may make the disk seem unresponsive even for reading. Difficult to say more without a glimpse at your script. – xhienne Aug 17 '17 at 15:37
  • 2
    What makes you think it's those tail/head commands that take a long time in your script? – Stéphane Chazelas Aug 17 '17 at 15:42
1

No, bash does not cache the output of commands, since the output can change from run to run; bash has no sure-fire way to track whether the file has been modified by another process, which would be critical, so that bash could tell whether its cached result was still valid or not.

However, there are other things at play here. When you use a shell (such as bash), you're simultaneously interacting with multiple parts of the system. For example:

  • The shell itself
  • GNU Readline, which is a line-editing interface used by multiple shells and other tools
  • The libc implementation, which can cause confusing behavior in nearly any program from time to time.
  • The terminal itself, which may have quirky behavior, and can actually respond to commands of its own. (E.g. either of Backspack and Delete may be unavailable, or they may be interchanged)
  • The GUI window the terminal exists inside of (if applicable). E.g. it may have an IME which allows typing Unicode characters into the terminal with a special key sequence (such as, Ctrl+Shift+u followed by a number.)
  • The kernel, including all of its modules and drivers
  • The hardware itself, which, of course, can overheat, short-circuit, loose power, etc

In this case, I'd say the biggest contributor is not bash itself, but the filesystem-level caching mechanism implement in the kernel. Once you've read a file once, a large chunk of it can end up in the filesystem cache, which is a hunk of memory reserved for this purpose.

When performing operations on the file a second time, the shell does not trigger a read of physical hardware again, but instead retrieves the file contents from the cache. Nearly any operation you (re)perform in bash will be extremely fast in comparison to a disk read. That's why you don't notice that bash is actually re-performing the commands, because the slow disk read is missing.

  • I wonder about this though, and maybe I am misunderstanding your explanation. I run the head/tail line from above on a file from the command line (on which I've not run it before and it takes almost a negligible amount of time. Files are the same size as each other, for testing purposes. I run the same command in the script and it takes 10-15x as long to finish. – Racehorse35 Aug 17 '17 at 16:56
  • @Racehorse35 In that case, I doubt very much that head or tail is to blame. Try to figure out which part of your script is actually slow, and that will lead you to the answer. – jpaugh Aug 17 '17 at 17:48
  • Incidentally, if you run the script on the same file more than once and it's still slow, then file caching is not playing an important role --- either because the file is too small or too big for it to matter. – jpaugh Aug 17 '17 at 17:49
  • Did you check out my edit? I think it is head/tails because there is an echo that should follow the heads and tails completion. Also, there is an echo that does print out right before. – Racehorse35 Aug 17 '17 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.