Get "Text Content" of Files

Question

I was wondering if there is a tool to scrape text content of a file. Basically I would like to put in a pdf, word document, compressed file or anything else and get the text it contains out.

There is strings which prints out sequences of printable characters but this doesn't work on many formats that are compressed. Its "dumb" nature also means that it often prints more than just the text. It would be a good fallback for an unknown file type though.

There is the pdftotext command which does pretty much exactly what I want but it only works on pdfs.

Is there a tool that works on multiple formats or that ties more of these tools together?

Additional Information

I lot of people has been mentioning that their tools are non-standard. I would just like to point out that this is fine.

Answer 1

Beyond strings I've never seen a Unix command line tool beyond grep, awk, and friends that can parse data from any text file. All these tools assume the data is in a textual format.

Files such as PDF, or Word documents encode the textual data in a binary format thus rendering it inaccessible without the aid of other tools to parse it out. These tools are typically one trick ponies that are often only able to deal with one or a few of these binary file formats.

You can use the file command to identify the file type.

Example

$ file /usr/share/cups/data/default.pdf
/usr/share/cups/data/default.pdf: PDF document, version 1.5

See man file for more on its usage.

Lucene, Solar, and other Indexing tools

You might be able to adapt or find a tool that's included with a indexing tool such as Lucene (such as: PDF & MSOffice docs) that can parse this text out of at least a large subset of the files one would typically deal with on a daily basis. I'd expect it to be able to contend with PDF, Word, and Libre Office file formats for starters.

Other tools

These tools can atleast partially read certain binary files so I"m adding them here just so you're aware of them. They aren't all encompassing but might be useful to you none the less.

• catdoc
• Swiss File Knife - A Command Line Tools Collection

Answer 2

No such conversion tool exists (at least not standard), but you might find some good preview generators or tools that tie existing ones together (see below). There are many, many document formats, and all of these formats are defined by the applications that work with them (random examples: PDFs, Word Documents, ODS spreadsheets, PowerPoint presentations, HTML documents [where some strings aren't part of content], JPG images of text, compiled programs, database files, zipped files, gzipped files, bzipped files, the list goes on and on). It would be virtually impossible to create a program that could read every document data format in existence, decode the data and pull out the actual content.

The typical strategy a search indexer / preview generator takes is to make a guess at the file type, invoke some filter than can grab relevant text, and process / preview that. Usually, this is accomplished by the author either writing a small set of filters for common types on his own (doing the research to learn about the format specifications), or by the application providing some type of extensible plugin-style framework and leaving it up to third-parties to write the filters. Many times it is a combination of both.

Nautilus takes a combination of these approaches, for example. For some formats, like common image formats, it can generate previews on its own. For other formats, it relies on third-party scripts, created by the authors of the applications and specified to Nautilus in gconf, that define various document formats, to parse documents and generate thumbnail images. It simply delegates the task of generating previews to these scripts, which are then stored as image files and displayed by Nautilus.

Any search indexer you find will operate on a similar principle.

You may want to research a few other options, including but certainly not limited to:

• Use of existing search software. Here is a good list and review of options available for Linux: http://www.tuxradar.com/content/best-linux-desktop-search-tools (if you have ADD: their top 3 in order are Recoll, Beagle, Google Desktop).
• If you are writing your own, consider writing one that can use already-written filters.
• Think of the document types you really want to support and write a script that uses a combination of file and whatever filter tools (e.g. pdftotext) work.
• Settle for limited uncompressed raw text and use strings.