49

Is there such a thing as a full-text indexing engine, that can be queried from the command line and ideally wouldn't require using a gui at all ?

I'm especially interested in indexing my ebooks and papers, so that's a mixture of pdf, epub and a few djvu. (Open)Office docs would be nice, but much lower on my list.

Improve this question
4
  • 1
    can you be more specific about what format the index takes?
    – ixtmixilix
    Commented Oct 13, 2010 at 23:49
  • Good point, edited.
    – julien
    Commented Oct 14, 2010 at 0:02
  • -1 As pointed out below Lucene or Tracker are good choices, but what's your problem with "use gtk, or even worse qt" ? Using just the command line is fine, but I don't see the point of criticizing a set of libraries...
    – tmow
    Commented Jan 27, 2011 at 8:16
  • 2
    @tmow: point taken, question edited. Didn't mean to undermine anyone's hard work, sorry if that's how it came through...
    – julien
    Commented Jan 27, 2011 at 11:24

7 Answers 7

Reset to default
12

Have you looked at Lucene or Sphinx? While you will need to initially parse the documents you want to index, once that's done, either one can search from the cli.

For Lucene, there is some info on doing this available.

Sphinx, is a bit more vague, but there is also some documentation available. You can pass structured XML data of your choice to sphinx via the xmlpipe2 data source.

Lucene relies on Java, while Sphinx is built in C++ with no needed outside dependencies.

Either one is going to require a bit of work to do what you want, but, seems like a totally workable solution.

Improve this answer
6
  • 1
    As an aside, if you want to index data that's in a DB (postrgres, mysql) then either of these also work incredibly well.
    – gabe.
    Commented Oct 14, 2010 at 3:14
  • hmm, I had dismissed sphinx for being too low-level, but looking at xmlpipe2 it seems a wrapper script for pdf2txt or the like would be pretty easy...
    – julien
    Commented Oct 14, 2010 at 13:02
  • +1 for mentioning in Lucene. I had good times with Lucene, it is a killer stuff!!
    – Nikhil Mulley
    Commented Jan 7, 2012 at 13:08
  • I believe elasticsearch is based on lucene, and might be more convenient to use (being a higher-level thing).
    – offby1
    Commented Jul 30, 2017 at 18:56
  • Fist link for lucene takes me to a confluence page that requires a login ... if Lucene is meant to be such a great tool, why is barrier to entry for it so high ??? i downloaded the software, ran luke.sh , all i want to do is pass a directory , create an index , then run a query, but instead i have to parse command line options , find syntax, add "unsupported" jdk modules, etc. Think i will look elsewhere ! I expected more from an apache project
    – niken
    Commented Apr 26, 2022 at 14:54
6

check out xapian. It has command line interface and can index a lot of formats.

Improve this answer
1
  • xapian is a great index, my personal favorite, and it's written in native C++!. Something which uses xapian would be most ideal, for many projects requiring indexing +1.
    – J. M. Becker
    Commented Oct 14, 2012 at 22:08
6

Recoll can be built with no GUI and will search your document types from the command line.

It uses Xapian under the hood.

Improve this answer
1
  • I've found recoll pretty good. tracker is an interesting option too.
    – Sridhar Sarnobat
    Commented Jun 11, 2022 at 5:08
5

This answer recommends using Google's codesearch,

Code Search is a tool for indexing and then performing regular expression searches over large bodies of source code.

Superusers on Debian/derivatives can try: sudo apt-get install codesearch

Improve this answer
1
  • 1
    this answer has some details on installing from source; in case it's not available in repos for your OS: superuser.com/a/1263343/65975
    – ccpizza
    Commented Feb 5, 2018 at 17:15
4

Tracker can be invoked from the command line and gtk+ is not a hard dependency for a project (but may be for packages).

Improve this answer
2
  • Well if I'm not mistaken, it'll still try to pull (large parts of) gnome as a dependency.
    – julien
    Commented Oct 13, 2010 at 23:05
  • 1
    As I said - project (in version 0.9.x at least the only GNOME hard dependency is glib). Packages may compiled the sane-default staff like GUI so you may need to compile it by hand.
    – Maja Piechotka
    Commented Oct 13, 2010 at 23:09
3

There currently are two streams of Tracker, stable (0.8) and unstable (0.9). Your OS likely has the 0.8 version, so if you can afford it (it has some bleeding edge software dependencies), go grab the latest tarfile (0.9.x). It has lots of improvements over 0.8, and is currently being stabilized further in order to be 0.10 (even numbers represent stability). If you choose to go this route, use this command to configure:

./configure --disable-tracker-needle --disable-tracker-preferences --disable-tracker-explorer --disable-tracker-status-icon

You are likely not going to have the dependencies installed, so it should be esier to simply install 0.8 from your distro, and just avoid the GUI bits. On Debian Squeeze, Ubuntu 10.10, and Ubuntu 11.04, these are nicely split. So (as root) run:

apt-get install --no-install-recommends tracker-utils tracker-miner-fs

The CLI tool for this is tracker-search, so run it with the --help option to see how to take advantage of it :-)

notes:

  • On Fedora 14, the Tracker package has dependencies on GTK+. I guess it's because it includes things like tracker-applet and tracker-preferences. They do however have a separate package for tracker-search-tool, the GUI search interface.
  • DjVu and ePUB aren't (yet) supported. Here's a list of of what is.
Improve this answer
1
  • Tracker is so refreshingly easy to use.
    – Sridhar Sarnobat
    Commented Jun 11, 2022 at 5:23
1

I worked on writing a full text search tool (a new apropos) for indexing and searching man pages for NetBSD this summer using Sqlite3. It consists of two command line tools:

  • makemandb: Which parses and builds an index of the content of man pages.
  • apropos: The tool for querying this index.

You could easily write a similar tool for yourself, for pdf's you will need a library for parsing pdf documents and similarly a utility to parse the open office documents.

You can read more about the project here

The code is here

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .