3

A while ago, I ran Recoll which created a database on a partition, so that I could quickly search among all the text information in the contents of all files (text file, pdf, djvu, html, etc) in that partition.

Now the partition, for which the Recoll created a database, has been wiped out accidentally, but I can still search in Recoll among those deleted files, because the database of Recoll is stored in another partition that hasn't been affected.

  1. When I search for a keyword among the text content of those deleted files, I can still see one or two lines surrounding that keyword as its context, It gives me some hope that it might be possible to recover the text content of each deleted file. But when I hit "Preview" or "Open" for a deleted file found by Recoll, Recoll will try to open the file that doesn't exist any longer, which of course fails. So I wonder if I can recover the deleted text files from Recoll's database?

  2. When I search for a keyword among the text content of those deleted files, I can also see the full path to each file found, So I wonder if I can recover the directory structure of my partition from Recoll's database?

Thanks and regards!

1 Answer 1

1

First, my condolences on the loss of your data. This answer is probably not useful to you nearly 8 years after the fact, but I'll answer it with the hope that it might be of use to someone else.

I wonder if I can recover the deleted text files from Recoll's database?

Yes, you can recover reconstructed document text, with some caveats.

I wonder if I can recover the directory structure of my partition from Recoll's database?

Yes, you can recover the file paths and from there reconstruct the directory structure, again with some caveats.

This can be accomplished via the xadump command, which is provided with recoll:

The xadump command is a low-level access and diagnostic tool for a Xapian index as organized by the Recoll indexer. The index directory to be used is specified with option -d.

Options -D, -X, -T and -r take a single docid argument specified with option -i. -D displays the document data record.

[ . . . ]

-r prints the document text as reconstructed from index data.

[ . . . ]

With option -q, xadump performs a simple AND query on the index, using the given term arguments.

https://www.lesbonscomptes.com/recoll/manpages/xadump.1.html

So, for example, to do a search for "independence", this command would work:

xadump -d ~/.recoll/xapiandb/ -q 'independence' | less

The first part of the query results looks like this for me:

DB: ndocs 100204 lastdocid 105155 avglength 7675.26
DB: terms are stripped
Performing query `Query(independence)'
Estimated results: 659

One of the results looks like this:

Document ID 89464 98% [url=file:///home/nathaniel/Dropbox/archive/2020/personal/projects/public-domain-documents/declaration-of-independence-html/index.html

mtype=text/html

fmtime=01585682999

origcharset=utf-8

fbytes=9365

pcbytes=9365

dbytes=8124

sig=93651585683000

caption= The Declaration of Independence of the United States of America

abstract=?!#@ THE DECLARATION OF INDEPENDENCE OF THE UNITED STATES OF AMERICA When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume, among the Powers of the

filename=index.html

]

So you can see the file:// URL with the path:

/home/nathaniel/Dropbox/archive/2020/personal/projects/public-domain-documents/declaration-of-independence-html/index.html

Of course, to reconstruct the full directory structure, you would have to do this for every document. It would probably be feasible to automate this, but would also be tricky and time-consuming to get it right. This also wouldn't help reconstruct the files that recoll didn't index.

We can reconstruct the document text with this command:

xadump -d ~/.recoll/xapiandb/ -i 89464 -r

Which gives this (I have truncated the third line for brevity):

DB: ndocs 100204 lastdocid 105155 avglength 7675.26

DB: terms are stripped

XP XPhome XPnathaniel XPDropbox XParchive XP2020 XPpersonal XPprojects XPpublic-domain-documents XPdeclaration-of-independence-html XCFNXXST XCFNindex.html XCFNhtml XCFNXXND SXXST the declaration of independence of the united states of america SXXND XSFNXXST index.html html XSFNXXND XXST the declaration of independence of the united states of america when in the course of human events it becomes necessary for one people to dissolve the political bands which have connected them with another and to assume among the powers of the earth the separate and equal station to which the laws of nature and of nature's s god entitle them a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation[...]

Here is the original HTML file:

<h1>
  THE DECLARATION OF INDEPENDENCE OF THE UNITED STATES OF AMERICA
</h1>
<p>
When in the Course of human events, it becomes necessary for one people to
dissolve the political bands which have connected them with another, and to
assume, among the Powers of the earth, the separate and equal station to
which the Laws of Nature and of Nature's God entitle them, a decent respect
to the opinions of mankind requires that they should declare the causes
which impel them to the separation.
</p>

Notice that this does contain the reconstructed text, but there are some issues:

  1. No capitalization. Everything is lowercase.

  2. No punctuation.

  3. No line-breaks. Everything is all on one line.

This isn't just HTML, either; here's part of the recovered text of the Project Gutenberg plain-text version of "Sense and Sensibility":

sense and sensibility by jane austen 1811 chapter 1 the family of dashwood had long been settled in sussex their estate was large and their residence was at norland park in the centre of their property where for many generations they had lived in so respectable a manner as to engage the general good opinion of their surrounding acquaintance

And here's the original text:

SENSE AND SENSIBILITY

by Jane Austen

(1811)




CHAPTER 1


The family of Dashwood had long been settled in Sussex.  Their estate
was large, and their residence was at Norland Park, in the centre of
their property, where, for many generations, they had lived in so
respectable a manner as to engage the general good opinion of their
surrounding acquaintance.

https://www.gutenberg.org/cache/epub/161/pg161.txt

Note that the output includes lastdocid 105155. With that in mind, here's a rough draft of what a script to dump the reconstructed text might look like:

#! /usr/bin/env bash

IMAX=105155
for ((i=1;i<=IMAX;i++))
do
    xadump -d ~/.recoll/xapiandb/ -i "$i" -r > "$i.txt"
done

Doing this for the first 100 documents took about 3 seconds on my machine, so for the full 100,000 documents it would probably require a little less than an hour.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .