1

Over a decade ago I imagined an improved version of grep that would use indexes to make recursive searching faster. I never got around to writing it, but I wonder if somebody else did!

My original idea was:

  • Keep a cached index of word -> [line-numbers] for each file, which would automatically update when the file size or timestamp changes.
  • Use the cache for lightning-fast searches over any files with up-to-date indexes, if the expression can make use of it.
  • For search expressions where the index cannot help, default to grep's usual behaviour.

Although this optimization would be limited to whole-word-searching, I thought it should be grep-compatible, so that it could be dropped in to replace /usr/bin/grep and help speed up various existing scripts that use grep.

There are limitations to this idea:

  1. Only word-searches or searches including certain \\<...\\> terms would be able to benefit from the indexes, and the latter case could be complex to implement.
  2. The time taken to check file stats to ensure the caches are up-to-date might make the time saved on searches minimal.

So do you think it would be worthwhile, or does point 2 make the idea worthless?

And has someone already tried this? If so, do please share.

3

1 Answer 1

3

Update 2019: I found csearch very useful when I was running on a HDD, which was relatively slow at searching hundreds of small files.

Now I am on a SSD, so reading is much faster, and I have switched to ripgrep which automatically respects .gitignore and avoids the need for an indexing step.

Both of them support regexp, but neither of them are 100% compatible with GNU grep.

Consider Google's csearch for a HDD

Google's Code Search: https://github.com/google/codesearch

Code Search is a tool for indexing and then performing regular expression searches over large bodies of source code.

It does not accept the same arguments as grep, notably it does not accept a file list and -r is assumed. However it does support a few of grep's favourite flags.

usage: csearch [-c] [-f fileregexp] [-h] [-i] [-l] [-n] regexp

So it can not act as a drop-in replacement for grep, but under certain conditions a call to grep could be converted into a call to csearch.

The regexp format (RE2, nearly PCRE) also differs from GNU grep. For example the patterns \< and \> cannot be used for start/end word; but instead we can use \b for word-boundary.

To install

On Debian and Debian clones:

apt-get install codesearch

For Ubuntu 12.04 (and other versions before saucy), download it by selecting your arch (i386/amd64) here and then install it with:

sudo dpkg -i ~/Downloads/codesearch_*.deb

To use

# Index the files
# It's fairly efficient but I wouldn't do it before every search
# I like to do this after switching branch
cd project
cindex .

# Search the index (case-insensitive)
csearch -i 'foo'

Global or local indexes

By default Code Search will index all projects into the ~/.csearchindex file, and any search you run will search across all projects.

If you want a separate index for a project, you can set the CSEARCHINDEX environment variable, for example:

export CSEARCHINDEX="$HOME/.csearchindex.projectx"

# Or to store the index in the current project's root folder
export CSEARCHINDEX="$(git rev-parse --show-toplevel)"/.csearchindex

Now cindex and csearch will use that local index file, instead of the global index.

0

Not the answer you're looking for? Browse other questions tagged .