Using bash
(which is not really a POSIX tool, but since you mentioned it explicitly):
#!/bin/bash
names=( *:* )
printf '%s\n' "${names[@]##*:}" | sort | uniq -c |
while read count hash; do
if [[ $count -gt 1 ]]; then
echo 'Would delete/move these:'
printf '%s\n' *:"$hash"
fi
done
This collects all names in the current directory that contains a :
character into the array names
. It is assumed that the pattern *:*
matches only the files we're interested in and that no other files have names like these.
The expansion of "${names[@]##*:}"
will result in a list of just the hashes, which we sort and count with sort | uniq -c
.
The result of that is read into count
and hash
in a while read
loop, and if the count is greater than one, we know that the hash is duplicated. If the hash is duplicated, the pattern *:"$hash"
would match all names that has that hash.
If you want to remove all files that have duplicated hashes, you may do
rm -f ./*:"$hash"
If you want to keep one of the files, then instead do, for example
dupnames=( ./*:"$hash" )
rm -f "${dupnames[@]:1}"
This sets the array dupnames
to the matching names, and deletes all but the first from the filesystem.
You may want to run with some debugging output enabled, and with the rm
disabled first, until you have convinced yourself that this actually works:
#!/bin/bash
names=( *:* )
printf '%s\n' "${names[@]##*:}" | sort | uniq -c |
while read count hash; do
if [[ $count -gt 1 ]]; then
echo 'Would delete/move these:'
dupnames=( ./*:"$hash" )
echo rm -f "${dupnames[@]:1}"
fi
done
A POSIX sh
variant of the above:
#!/bin/sh
for name in *:*; do
printf '%s\n' "${name##*:}"
done | sort | uniq -c |
while read count hash; do
if [ "$count" -gt 1 ]; then
echo 'Would delete/move these:'
set -- ./*:"$hash"
shift
echo rm -f "$@"
fi
done
A variation on this last one that does away with sort | uniq -c
by means of awk
:
#!/bin/sh
for name in *:*; do
printf '%s\n' "${name##*:}"
done |
awk ' { count[$0]++ }
END { for (hash in count) if (count[hash] > 1) print hash }' |
while read hash; do
echo 'Would delete/move these:'
set -- ./*:"$hash"
shift
echo rm -f "$@"
done
The awk
snippet could also replace sort | uniq -c
in the other pieces of code in this answer, but note that the final loop now does not need to test whether the count is greater than one, and that it only reads the hashes.