Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have noticed that subsequent runs of grep on the same query (and also a different query, but on the same file) are much faster than the first run (the effect is easily noticeable when searching through a big file).

This suggests that grep uses some sort of caching of the structures used for search, but I could not find a reference on the Internet.

What mechanism enables grep to return results faster in subsequent searches?

share|improve this question
add comment

3 Answers

up vote 45 down vote accepted

Not grep as such, but the filesystem itself often caches recently read data, causing later runs to go faster since grep is effectively searching in memory instead of disk.

share|improve this answer
2  
I concur. First time you grep a file, the filesystem code loads the file from disk into cache; when you grep it again shortly afterwards, it hasn't had time to expire from the cache yet. See if there's a way to manually flush the filesystem cache, then compare before and after times. –  Shadur Mar 9 '11 at 10:56
2  
@Shadur — there is a way. See unix.stackexchange.com/questions/8398/… –  mattdm Mar 9 '11 at 16:07
2  
There's no practical difference, but it's a kernel cache, not the filesystem. –  pboin Mar 9 '11 at 18:40
    
@pboin Ok, I took a small shortcut. Of course it is the part of the kernel that handles the filesystem that does the caching not the filesystem (on the disk) in itself. Pedantic but true. :) –  PaulRein Mar 10 '11 at 14:49
add comment

Linux and *NIX systems make use of various caches which sit between the filesystems (abstracted through VFS) and the user level processes. So it's not grep and it's not the filesystem doing the caching - it's the operating system.

The cache which is responsible for your grep performance is the VFS Buffer Cache. Other caches are for inodes and directories but those wouldn't come into play here.

For more info see: Linux Documentation Project: 9.2 The Virtual File System http://tldp.org/LDP/tlk/fs/filesystem.html

share|improve this answer
5  
If you're doing a recursive grep, the inode and dentry caches make a noticeable difference. In a test, a grep within /usr/include took ~16.4 seconds on my system with all caches dropped. With everything cached, about 0.3 seconds. If I drop just the page cache (which holds the buffer cache), it takes ~14.3 seconds. If I leave the page cache and drop the dentry/inode caches, it's about 12 seconds. –  mattdm Mar 9 '11 at 16:07
    
@mattdm, that's interesting. How did you drop the caches? –  JRW Mar 9 '11 at 16:35
    
add comment

And if you are looking for a cached version of grep, check out http://stackoverflow.com/questions/7734596/grep-but-indexable

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.