2

Is there a tool that allows to index (search) files based on their metadata on linux? I've search a bit and found (here) that there are several tools that allow file indexing on linux:

But it seems like none of these indexes files' metadata (or maybe I just haven't found how in the documentation). This may look as an answer, but the documentation is quite short and I can't figure if that's really what I'm looking for.

I found this: GWorkspace, it doesn't look very legit but it states that it does allow metadata based indexing:

GWMetadata is the metadata indexing and search system which includes the live folders functionality.

My goal

To be a bit more precise, I would like to mimic what can (natively) be done on macOs:

# Create a file and set metadata
touch test_file.txt
xattr -w com.apple.metadata:MY_META todo test_file.txt
# Wait a few seconds for the index to update
sleep 5
# Search files that have a given metadata
mdfind "MY_META=todo"

Which returns the path of the file we just created: $(pwd)/test_file.txt. I'm looking for a tool that has a CLI interface because I need to automate things around these few commands.

2
  • Yes if it makes a difference bash would be juste fine.
    – cglacet
    Jun 26, 2022 at 9:58
  • As far as I understand, baloo ( community.kde.org/Baloo ) should be capable of such a thing. (including CLI) Taking into account tons of limitations not to mention that the first optimization that many, if not all, wise kde-plasma users, carry on consists in : disabling it.
    – MC68020
    Jun 29, 2022 at 19:09

1 Answer 1

1

I don't know of a similar facility on Linux (Debian-flavored ) or BSD hosts, but on a system where xattr is enabled, rolling your own is not overly complicated. I give a working solution at the end of this answer in the form of a POSIX compliant script for searching (not for indexing per se). I integrated a CLI usage help functionality, so getting to the stage of using it should not require poring over every detail of the code, although "read the code" is the best advice overall.

Indexing would require running the script unchanged but as a daemon, perhaps as a systemd service unit, and updating a small database (which implies persisting pertinent data in a table of sorts). Doing so on a continuous basis however, would place a significant burden on the indexed FS's host, as in Baloo's case on KDE for instance. Short of a compelling reason to do so, I would avoid it altogether and stick to a much lighter-weight CLI spot-search utility instead, such as the one proposed here.


Script description: findxattr

  • is based on the well known find external utility and its idiom:
    find [options, filters ...] -exec sh -c '[...]' sh_exec "{}" \; 2> dev/null

  • runs in POSIX-compliant mode even on hosts where the shebang #!/usr/bin/sh points to a fake Bourne shell, meaning that you will often find that
    /usr/bin/sh --> /usr/bin/bash.

  • consists of a find-wrapper. It does not implement every bells and whistles commonly available in find implementations, but it complies with OP's requirements and goes beyond to offer a little more flexibility to ClI users interested in refining their searches. In particular, it implements:

    • a "help" mode, by issuing findxattr -h|--help, that documents correct usage,
    • long and short options, preceded respectively by one and two hyphens,
    • two filters -p|--path and -m|--md|--maxdepth, very similar to
      find [-maxdepth <d>] [-path <path_specs>] ... (see man find)
    • global searches from $PWD (when invoked with no argument, or with -a|--all),
    • tagged searches by x-attribute's KV properties, namely:
      • key -x|--xattr <xattr_name>==,
      • value -x|--xattr ==<xattr_value>,
      • both -x|--xattr <xattr_name>==<xattr_value>,

    where x-attributes' names and values can include spaces if adequately quoted.

  • can be trivially expanded to include more options and switches that getopt (in the script) will gladly parse. To exploit additional options and filters, expanded logic is needed, but the way existing options are treated is a blueprint for expanding the tool's scope.


Script testing:
First build some toy x-attributes.

$ touch ~/fully/qualified/foo ~/fully/qualified/_1_path/bar ~/fully/qualified/_2_path/foobar ~/fully/qualified/baz

$ setfattr -n user.likes -v 'I like strawberries.' ~/fully/qualified/foo

In the above, with setfattr, user.__ points toward the user namespace for the x-attribute named likes. New x-attributes are defined as key-value (KV) pairs. Here keys are the strings likes, dislikes, filebirth and their values must be preceded by -v.

Using attr instead of setfattr (and replacing -v by -V), x-attributes are always created in user namespace by default:

$ cd ~/fully/qualified
$ attr -s dislikes -V 'hacks' ./foo
$ attr -s filebirth -V '20220627-193029-CEST' ./_1_path/bar
$ attr -s filebirth -V '20210330-185430-CEST' ./_2_path/foobar
$ attr -s dislikes -V 'java' ./baz

To set, get, remove or list extended attributes, you can also use attr on compatible filesystem objects (see man attr):

Usage:
attr [-LRSq] -s attrname [-V attrvalue] pathname # to set value
attr [-LRSq] -g attrname pathname # to get value
attr [-LRSq] -r attrname pathname # to remove attr
attr [-LRq] -l pathname # to list attrs
-s reads a value from stdin and -g writes a value to stdout

So, for instance:

$ cd ~/fully/qualified

$ attr -qg likes ./foo
I like strawberries.
$ attr -qg filebirth ./_1_path/bar
20220627-193029-CEST

The script produces a tab-separated output, e.g.:

$ cd ~; pwd
/home/USER

$ findxattr -m 4                             # search entire subtree (from `$PWD`) with max depth of 4    
./fully/qualified/foo        likes        I like strawberries.
./fully/qualified/foo        dislikes     hacks
./fully/qualified/_1_path/bar        filebirth        20220627-193029-CEST
./fully/qualified/_2_path/foobar        filebirth        20210330-185430-CEST
./fully/qualified/baz        dislikes     java

$ findxattr -m 3 -x dislikes==               # search entire subtree (from `$PWD`) by name, depth
./fully/qualified/foo        dislikes     hacks
./fully/qualified/baz        dislikes     java

$ findxattr -m 4 -p "*lified/_2_*" -x filebirth== # search by depth, path, name
./fully/qualified/_2_path/foobar        filebirth        20210330-185430-CEST

$ findxattr -m 4 -x =='20220627-193029-CEST' # search entire subtree (from `$PWD`) by value, depth
./fully/qualified/_1_path/bar        filebirth        20220627-193029-CEST

$ findxattr -x filebirth==                   # search entire subtree (from `$PWD`) by name
./fully/qualified/_1_path/bar        filebirth        20220627-193029-CEST
./fully/qualified/_2_path/foobar     filebirth        20210330-185430-CEST

Code:

#!/usr/bin/sh

#-------------------------------------------------------------
# Author: CBhihe
# Copyright (c) 2022 C. K. Bhihe
# Available under GPL3 at github.com/Cbhihe
#-------------------------------------------------------------

version='0.5.0'

set -o posix
#set -x

getopt_parsed=$(getopt -q -s sh -a -l 'all,help,path:,xattr:,maxdepth:' -o '+ham:p:x:' -- "$@")
exit_code=$?
if [ $exit_code -ne 0 ]; then
    printf "getopt exited with code %d (%s).\n%s\n" $exit_code "getopt parsing error"\
    "The most probable cause is an unknown option. Review command usage with \`findxattr -h|--help'" >&2
    exit 1
fi

eval set -- "$getopt_parsed"
unset getopt_parsed

xattrkey=""
xattrval=""

while true; do
    case "$1" in
        '-a'|'--all')
            /usr/bin/find . -exec sh -c '
                while IFS= read -r xattrkey; do
                    xattrvalout=`/usr/bin/attr -qg "$xattrkey" "$1" 2>/dev/null`
                    printf "%-20s\t%-20s\t%s\n" "$1" "$xattrkey" "$xattrvalout"
                done < <(/usr/bin/attr -ql "$1" 2>/dev/null)
            ' sh_exec "{}" \; 2>/dev/null
            exit 0
        ;;

        '-m'|'--maxdepth'|'--md'|'--maxd')
            tmp=`expr "$2" + 0`
            if (( "$2" == "$tmp" )); then
                maxdepth="$2"
                shift 2
                continue
            else
                printf "The script exited because '--maxdepth' arg must be a positive integer; current arg is %s.\n%s\n" "$2" "Review command usage with \`findxattr -h|--help'" >&2
                exit 1
            fi
        ;;

        '-p'|'--path')
            locpath="$2"
            #if [ "$locpath" = "\(/*[^/]\+\)\+/$" ]; then
            #if expr "$locpath" : "^\(/*\([^/]\)\+\)\+$" >/dev/null; then
            if expr "$locpath" : "^\(/*[^/]\+\)\+$" >/dev/null; then
                shift 2
                continue
            else
                printf "The script exited because '--path' arg must be a valid path; current arg is %s.\n%s\n" "$2" "Review command usage with \`findxattr -h|--help'" >&2
                exit 1
            fi
        ;;

        '-x'|'--xattr')
            keyval="$2"
            found=1
            if [ "$keyval" != "${keyval%==*}" ]; then
                found=0
            fi

            if [ "$found" = "1" ]; then
                printf "The script exited because '-x|--xattr' arg appears to be either empty or malformed.\nReview command usage with \`findxattr -h|--help'. Remember that extended attributes\ncan be sought by key AND value (<key>==<value>), or only by key (<key>==), or only by value\n(==<value>), where in each case the parenthesized content represents the '-x' option's argument.\n" >&2
                exit 1
            else
                xattrkey="${keyval%==*}"
                xattrval="${keyval#*==}"
                shift 2
                continue
            fi
        ;;

        '-h'|'--help')
            printf "%s\n" " " "This is a script based on \`find' but restricted to the options shown below. Both short- and long-" \
"format options are allowed. Unknown options cause the script to abort with exit code 1." \
" " \
"Usage:" \
"   \`findattr -h|--help' Prints this usage information. This option is used alone." \
"   \`findattr -a|--all'  Searches recursively for all files with xattr(s) starting at \$PWD." \
"                        This option is used alone." \
"   \`findattr [-m|maxdepth <d>] [-p|path <path>] [-x|xattr <xattr_name>==<xattr_value>]'" \
"       Options that can be combined:" \
"         -m|--maxdepth  Identical to \`find -maxdepth <d>' option, where \`d' a positive integer;" \
"                        Limits any recursive search with \`find', to the specified level of the file tree." \
"                        Note that the level of the file tree is understood counting from \$PWD, NOT" \
"                        from a supposed start point represented by the \`--path' argument if present." \
"         -p|--path      Identical to \`find -path <spath>' option;" \
"                        Traverse the file tree from the specified path, searching for ANY xattr," \
"                        unless the \`--xattr' option is invoked to filter the search so a specific xattr" \
"                        name and value can be sought." \
"         -x|--xattr     Lists files with specified \`xattr', n the file tree starting at \$PWD unless" \
"                        \`--path' is invoked." \
"                        A compulsory argument of the form: '<xattr_name>==<xattr_value>' is expected." \
"                        Quoting is needed in case the argument contains space(s) or special characters." \
" "
            exit 0
        ;;

        '--')
            shift
            break
        ;;

        *)
            printf "%s\n" "Internal error. Abort." >&2
            exit 1
        ;;

    esac
done

if [ -n "$maxdepth"  ] && [ -n "$locpath" ]; then
    set -- -maxdepth "$maxdepth" -path "$locpath"
elif [ -z "$maxdepth"  ] && [ -n "$locpath" ]; then
    set -- -path "$locpath"
elif [ -z "$maxdepth"  ] && [ -z "$locpath" ]; then
    set --
else
    #[ -n "$maxdepth"  ] && [ -z "$locpath" ]
    set -- -maxdepth "$maxdepth"
fi

if [ -n "$xattrkey" ] && [ -n "$xattrval" ]; then
#if expr "$xattrkey" != "" >/dev/null && expr "$xattrval" != "" >/dev/null; then
    xattrkey="$xattrkey" xattrval="$xattrval" /usr/bin/find . "$@" -exec sh -c '
        xattrvalout=`/usr/bin/attr -qg "$xattrkey" "$1" 2>/dev/null`
        if [ "$xattrvalout" = "$xattrval" ]; then
        #if expr "$xattrvalout" = "$xattrval" >/dev/null; then
            printf "%-20s\t%-20s\t%s\n" "$1" "$xattrkey" "$xattrvalout"
        fi
    ' sh_exec "{}" \; 2>/dev/null

elif [ -n "$xattrkey" ] && [ -z "$xattrval" ]; then
#elif expr "$xattrkey" != "" >/dev/null && expr "$xattrval" = "" >/dev/null; then
    xattrkey="$xattrkey" xattrval="$xattrval" /usr/bin/find . "$@" -exec sh -c '
        xattrvalout=`/usr/bin/attr -qg "$xattrkey" "$1" 2>/dev/null`
        if [ -n "$xattrvalout" ]; then
            while IFS= read -r xattrvalout || [ -n "$xattrvalout" ]; do
                printf "%-20s\t%-20s\t%s\n" "$1" "$xattrkey" "$xattrvalout"
            done <<<"$xattrvalout"
        fi
    ' sh_exec "{}" \; 2>/dev/null

elif [ -z "$xattrkey" ] && [ -z "$xattrval" ]; then
#elif expr "$xattrkey" = "" >/dev/null && expr "$xattrval" = "" >/dev/null; then
    /usr/bin/find . "$@" -exec sh -c '
        xattrkeys=`/usr/bin/attr -ql "$1" 2>/dev/null`
        if [ -n "$xattrkeys" ]; then
            while IFS= read -r xattrkey || [ -n "$xattrkey" ]; do
                xattrvalouts=`/usr/bin/attr -qg "$xattrkey" "$1" 2>/dev/null`
                if [ -n "$xattrvalouts" ]; then
                    while IFS= read -r xattrvalout || [ -n "$xattrvalout" ]; do
                        printf "%-20s\t%-20s\t%s\n" "$1" "$xattrkey" "$xattrvalout"
                    done <<<"$xattrvalouts"
                fi
            done <<<"$xattrkeys"
        fi
    ' sh_exec "{}" \; 2>/dev/null

else
    # [ -z "$xattrkey" ] && [ -n "$xattrval" ]; then
    # expr "$xattrkey" = "" >/dev/null && expr "$xattrval" != "" >/dev/null
    xattrkey="$xattrkey" xattrval="$xattrval" /usr/bin/find . "$@" -exec sh -c '
        xattrkeys=`/usr/bin/attr -ql "$1" 2>/dev/null`
        if [ -n "$xattrkeys" ]; then
            while IFS= read -r xattrkey || [ -n "$xattrkey" ]; do
                xattrvalouts=`/usr/bin/attr -qg "$xattrkey" "$1" 2>/dev/null`
                if [ -n "$xattrvalouts" ]; then
                    while IFS= read -r xattrvalout || [ -n "$xattrvalout" ]; do
                        if [ "$xattrvalout" = "$xattrval" ]; then
                            printf "%-20s\t%-20s\t%s\n" "$1" "$xattrkey" "$xattrvalout"
                        fi
                    done <<<"$xattrvalouts"
                fi
            done <<<"$xattrkeys"
        fi
    ' sh_exec "{}" \; 2>/dev/null

fi

exit 0
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .