How to associate a file to an hash code, and use it to find the file anywhere in the filesystem

Question

I would like to create a tool (a shell script, or maybe a Python script) to generate and associate an hash code to a file, and then be able to use that hash to retrieve the file in the filesystem (open the parent folder and highlight the file, and/or open the file with the default application).

I am used to something similar, because I was using BibDesk, a database software to manage scientific articles, that runs only under MacOS. BibDesk uses a similar method to link the pdf files to their entries in the database, so that the association still works if you change the name to the file, or move it to another location in the filesystem.

See also this answer to a related question.

Answer 1

Of course, I can't look inside BibDesk. However, by its functional description I'd say the main job it does is keep a database. In that database it would associate hashes with files.

It would then watch the folders it's supposed to, and look for files with changes. Considering even large personal literature databases will not have millions of files, even a rescan to verify the hashes of the files it finds are still as expected, would hardly be noticeable, especially if done in the background.

The files on the file systems on your computer are path-adressed, not content or hash-addressed – every additional lookup information needs to be stored separately. (You can store additional information about files in most filesystems, but to look up that information you would need to know the path of the file – hence not solving your problem.)

So, your answer over there is a tiny bit misleading – you can't use a file's content hash to retrieve a file from a filesystem. (you can of course change the name of the file to be the hash – but that's not what you meant, I think.)

However, keeping the hash in the database might be good idea for integrity reasons (you can check the hash is correct before delivering the file), and as you said, if you can afford tracking file changes, or frequent rescans, for re-discovery reasons.

Answer 2

You can do this, either in python or bash.

A hash/value database could be used to store the hash and values. Take a search for gdbm tools and libraries. The original tool originated on BSD unix, gdbm is the gnu version and they are generally not distributed in the standard install these days.

I installed gdbmtool sudo apt install gdbmtool on ubuntu (20.04) to get a utility to help me create the hash/value pairs db using bash. I sued sha256sum again a command line utility to generate the hash on the files.

My aim was different to yours, since I wanted a tool to detect photo's (and then music) that were identical but that I had renamed in the past so that I could remove duplicates, I also had them all located under one directory so I wasn't searching across the entire filesystem. I started with a tool called dedupe but I found that cumbersome for my needs. This was some time back, there are other tools to do the same these days (fslint, dupeguru,...).

In addition to the initial creation of the database, you will need something that is run regularly to update the db when the files are moved (renamed) to set the new path value, to the correct new location and to add new entries when they appear.

I don't think you want to hash the file name paths, but rather hash the contents of the file so that you can detect an existing identical file found in a new location.

Once you have the path of the file, it can be opened using xdg-open from the command line (or your script).

A relational database, perhaps even sqlite might be more appropriate, if you are wanting to track multiple paths against the same hash, because that is not allowed key-value db. But I am venturing slightly off-topic (Unix & Linux) if I go further, I have given you the names of some Unix tools to search and read up on, to get you going.

Answer 3

If I understood your question right, you could achieve this with user xattrs. Install attr package in distro you use. For example in debian/ubuntu with command sudo apt install attr. Then reand man attr, man setfattr and man getfattr Then you could save hash in files extended attributes with command like this.

setfattr -n user.md5sum -v $(md5sum test.bin | awk '{print $1}') test.bin

And read with command

getfattr -n user.md5sum  test.bin
# file: test.bin
user.foo="a7fd41d58563137a6f73e738008d9970"

Notice than attribute name must be prefixed with namespace user.