41

I have noticed that subsequent runs of grep on the same query (and also a different query, but on the same file) are much faster than the first run (the effect is easily noticeable when searching through a big file).

This suggests that grep uses some sort of caching of the structures used for search, but I could not find a reference on the Internet.

What mechanism enables grep to return results faster in subsequent searches?

Improve this question

3 Answers 3

Reset to default
63

Not grep as such, but the filesystem itself often caches recently read data, causing later runs to go faster since grep is effectively searching in memory instead of disk.

Improve this answer
4
  • 2
    I concur. First time you grep a file, the filesystem code loads the file from disk into cache; when you grep it again shortly afterwards, it hasn't had time to expire from the cache yet. See if there's a way to manually flush the filesystem cache, then compare before and after times.
    – Shadur
    Mar 9, 2011 at 10:56
  • 3
    @Shadur — there is a way. See unix.stackexchange.com/questions/8398/…
    – mattdm
    Mar 9, 2011 at 16:07
  • 3
    There's no practical difference, but it's a kernel cache, not the filesystem.
    – pboin
    Mar 9, 2011 at 18:40
  • @pboin Ok, I took a small shortcut. Of course it is the part of the kernel that handles the filesystem that does the caching not the filesystem (on the disk) in itself. Pedantic but true. :)
    – PaulRein
    Mar 10, 2011 at 14:49
19

Linux and *NIX systems make use of various caches which sit between the filesystems (abstracted through VFS) and the user level processes. So it's not grep and it's not the filesystem doing the caching - it's the operating system.

The cache which is responsible for your grep performance is the VFS Buffer Cache. Other caches are for inodes and directories but those wouldn't come into play here.

For more info see: Linux Documentation Project: 9.2 The Virtual File System http://tldp.org/LDP/tlk/fs/filesystem.html

Improve this answer
3
  • 5
    If you're doing a recursive grep, the inode and dentry caches make a noticeable difference. In a test, a grep within /usr/include took ~16.4 seconds on my system with all caches dropped. With everything cached, about 0.3 seconds. If I drop just the page cache (which holds the buffer cache), it takes ~14.3 seconds. If I leave the page cache and drop the dentry/inode caches, it's about 12 seconds.
    – mattdm
    Mar 9, 2011 at 16:07
  • @mattdm, that's interesting. How did you drop the caches?
    – JRW
    Mar 9, 2011 at 16:35
  • unix.stackexchange.com/questions/8398/… :)
    – mattdm
    Mar 9, 2011 at 17:09
1

And if you are looking for a cached version of grep, check out https://stackoverflow.com/questions/7734596/grep-but-indexable

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .