Delete all files except one matching a given wildcard pattern in a directory

Question

I have many files in a directory, and I want to remove all but one of the files that have the same prefix. For example, I have the files with the pattern filename.__<random_string>.pdf, (filename can be any string of some length)

foo.__.pdf
foo.__resume.pdf 
foo.__name.pdf
bar.__.pdf
bar.__resume.pdf
bar.__name.pdf

Now from them I only want one of the three files which have the same prefix, i.e, I only want either of the first three files and either one of the last three. For example, the directory should contain,

foo.__.pdf
bar.__.pdf

Answer with any of the scripting language or shell is accepted.

Answer 1

#!/bin/bash

declare -A seen

for name in *.__*.pdf; do
        prefix=${name%%.__*.pdf}

        if [[ -z ${seen[$prefix]} ]]; then
                printf 'keeping "%s"\n' "$name"
                seen[$prefix]=1
        else
                printf 'deleting "%s"\n' "$name"
                # rm -f -- "$name"
        fi
done

The script above extracts the prefix from each filename matching the filename globbing pattern *.__*.pdf in the current directory. If the prefix has not been seen before, the file is kept. Otherwise the file is deleted (the rm command is currently commented out for safety).

To track what prefixes have been seen, they are stored as keys in an associative array called seen. Associative arrays were introduced in bash release 4.

---

Since any file matching *.__*.pdf with the same prefix is "equivalent", just renaming all those files to the same name would reduce them down to a single file.

This does not require an associative array and can easily be done by /bin/sh:

#!/bin/sh

for name in *.__*.pdf; do
        prefix=${name%%.__*.pdf}

        printf 'moving "%s" to "%s.__.pdf"\n' "$name" "$prefix"
        # mv -f -- "$name" "$prefix.__.pdf"
done

Here, all files with the prefix foo are moved to the name foo.__.pdf (the mv command is commented out for safety).

Answer 2

With zsh:

all=(*.__*.pdf)
typeset -A hash
for f ($all) hash[${f%%.__*}]=$f
keep=($hash)
rm -f -- ${all:|keep}

Among the ones with same prefix, that will keep the one that comes last in lexical order. You can reverse that order with all=(*.__*.pdf(On)) instead of all=(*.__*.pdf).

Answer 3

With zsh:

declare -A h=()
rm -- *(e:'((h[${REPLY%%.*}]++))':)

Unfortunately, this doesn't allow setting the order. Adding the o/O glob qualifiers here would only alter the order after excluding the first of each, and e evaluates whether to include/exclude a file in the order the files are found (i.e. unordered as *(oN) would return them).

Answer 4

Using gawk:

ls | awk '/\.__.*.pdf/{a=$0; 
sub(/__.*.pdf/, "",a); 
f=sprintf("%s__.pdf",a); 
if (f!=$0) print "rm", $0}'  

For printing files that are desired, the following would do

ls | awk '/\.__.*.pdf/{a=$0; 
sub(/__.*.pdf/, "",a); 
f=sprintf("%s__.pdf",a); 
if (f==$0) print ; else print "rm", $0}'

Answer 5

Using perl we can open a directory handle on the current directory and read off all plain files whose names match the specified criterion. The first of those files having the same prefix have their names printed to stdout whilst the subsequent ones with that prefix are deleted. The prefixes are made keys of a hash %h

perl -le '
  opendir my $dh, "."
    or die qq(Cannot opendir ".": $!);

  -f && /^(.+?)\.__.*\.pdf$/ and
    !$h{$1}++ ? print() : unlink()
  while readdir $dh;

  closedir $dh;
'

---

Using find command:

fx() {
  set -- "$1" "$tmpdir/${1%%.__*.pdf}"
  if [ ! -s "$2" ]; then
    echo x > "$2"
    shift "$#"
  else
    rm -f "$1"
  fi
  return "$#"
}
export -f fx
tmpdir="$(mktemp -d)" \
find . ! -name . -prune -type f \
-name '?*.__*.pdf' \
-exec sh -c 'fx "$1"' find-sh {} \; -print;

---

Using python3 along with the pathlib module with the same idea as above:

python3 -c 'from pathlib import Path
seen = {}
for f in Path(".").glob("?*.__*.pdf"):
  if f.is_file():
    p = f.name.find(".__")
    k = f.name[0:p]
    if p > 0:
      if k in seen: f.unlink()
      else:
        print(f.name)
        seen[k]=1
'

Answer 6

You may use a combination of find and awk:

 find ./ -mindepth 1 -maxdepth 1 -type f -print0 |
 awk 'BEGIN {RS="\x00" ; FS="."}
      seen[$2]++ {system("rm '\''"$0"'\''")}'

This uses NUL as file separator so also can handle things like newlines in names. Assumes that . is used for the prefix separator! If you want the first (by dictionary order) of each batch, you may also pipe find's results through sort:

 find ./ -mindepth 1 -maxdepth 1 -type f -print0 | sort -z |
 awk ....

For exactly matching your pattern .__, you may as well use:

awk 'BEGIN {RS="\x00" ; FS=".__" } seen++[$1] { ... }'