How to partially match only one instance of a filename?

Question

So i have a list of files, that i renamed to filename:hash.

What i wanted to do is to only match the hash, while keeping the combo filename:hash intact, and not calculating their hash again, since the files didn't change.

While doing this, i need to either move them, or remove them, but since the filename would make it not "unique" enough for the tool uniq, this won't work if directly using a pipe.

is there any way to do this? without using any other tools than posix tools like awk, bash etc, and without using a list or DB file?

details: No, this isn't technically a duplicate to this post, and Yes, the end goal is technically the same(which is to remove or move the duplicate using the method/in the situation i already described on the other post, or here).

Answer 1

Using bash (which is not really a POSIX tool, but since you mentioned it explicitly):

#!/bin/bash

names=( *:* )

printf '%s\n' "${names[@]##*:}" | sort | uniq -c |
while read count hash; do
    if [[ $count -gt 1 ]]; then
        echo 'Would delete/move these:'
        printf '%s\n' *:"$hash"
    fi
done

This collects all names in the current directory that contains a : character into the array names. It is assumed that the pattern *:* matches only the files we're interested in and that no other files have names like these.

The expansion of "${names[@]##*:}" will result in a list of just the hashes, which we sort and count with sort | uniq -c.

The result of that is read into count and hash in a while read loop, and if the count is greater than one, we know that the hash is duplicated. If the hash is duplicated, the pattern *:"$hash" would match all names that has that hash.

If you want to remove all files that have duplicated hashes, you may do

rm -f ./*:"$hash"

If you want to keep one of the files, then instead do, for example

dupnames=( ./*:"$hash" )
rm -f "${dupnames[@]:1}"

This sets the array dupnames to the matching names, and deletes all but the first from the filesystem.

You may want to run with some debugging output enabled, and with the rm disabled first, until you have convinced yourself that this actually works:

#!/bin/bash

names=( *:* )

printf '%s\n' "${names[@]##*:}" | sort | uniq -c |
while read count hash; do
    if [[ $count -gt 1 ]]; then
        echo 'Would delete/move these:'
        dupnames=( ./*:"$hash" )
        echo rm -f "${dupnames[@]:1}"
    fi
done

A POSIX sh variant of the above:

#!/bin/sh

for name in *:*; do
    printf '%s\n' "${name##*:}"
done | sort | uniq -c |
while read count hash; do
    if [ "$count" -gt 1 ]; then
        echo 'Would delete/move these:'
        set -- ./*:"$hash"
        shift
        echo rm -f "$@"
    fi
done

A variation on this last one that does away with sort | uniq -c by means of awk:

#!/bin/sh

for name in *:*; do
    printf '%s\n' "${name##*:}"
done |
awk '    { count[$0]++ }
     END { for (hash in count) if (count[hash] > 1) print hash }' |
while read hash; do
    echo 'Would delete/move these:'
    set -- ./*:"$hash"
    shift
    echo rm -f "$@"
done

The awk snippet could also replace sort | uniq -c in the other pieces of code in this answer, but note that the final loop now does not need to test whether the count is greater than one, and that it only reads the hashes.