35

I have noticed that subsequent runs of grep on the same query (and also a different query, but on the same file) are much faster than the first run (the effect is easily noticeable when searching through a big file).

This suggests that grep uses some sort of caching of the structures used for search, but I could not find a reference on the Internet.

What mechanism enables grep to return results faster in subsequent searches?

59

Not grep as such, but the filesystem itself often caches recently read data, causing later runs to go faster since grep is effectively searching in memory instead of disk.

  • 2
    I concur. First time you grep a file, the filesystem code loads the file from disk into cache; when you grep it again shortly afterwards, it hasn't had time to expire from the cache yet. See if there's a way to manually flush the filesystem cache, then compare before and after times. – Shadur Mar 9 '11 at 10:56
  • 3
    @Shadur — there is a way. See unix.stackexchange.com/questions/8398/… – mattdm Mar 9 '11 at 16:07
  • 3
    There's no practical difference, but it's a kernel cache, not the filesystem. – pboin Mar 9 '11 at 18:40
  • @pboin Ok, I took a small shortcut. Of course it is the part of the kernel that handles the filesystem that does the caching not the filesystem (on the disk) in itself. Pedantic but true. :) – PaulRein Mar 10 '11 at 14:49
19

Linux and *NIX systems make use of various caches which sit between the filesystems (abstracted through VFS) and the user level processes. So it's not grep and it's not the filesystem doing the caching - it's the operating system.

The cache which is responsible for your grep performance is the VFS Buffer Cache. Other caches are for inodes and directories but those wouldn't come into play here.

For more info see: Linux Documentation Project: 9.2 The Virtual File System http://tldp.org/LDP/tlk/fs/filesystem.html

  • 5
    If you're doing a recursive grep, the inode and dentry caches make a noticeable difference. In a test, a grep within /usr/include took ~16.4 seconds on my system with all caches dropped. With everything cached, about 0.3 seconds. If I drop just the page cache (which holds the buffer cache), it takes ~14.3 seconds. If I leave the page cache and drop the dentry/inode caches, it's about 12 seconds. – mattdm Mar 9 '11 at 16:07
  • @mattdm, that's interesting. How did you drop the caches? – JRW Mar 9 '11 at 16:35
1

And if you are looking for a cached version of grep, check out https://stackoverflow.com/questions/7734596/grep-but-indexable

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.