I have many files in a directory, and I want to remove all but one of the files that have the same prefix. For example, I have the files with the pattern filename.__<random_string>.pdf, (filename can be any string of some length)


Now from them I only want one of the three files which have the same prefix, i.e, I only want either of the first three files and either one of the last three. For example, the directory should contain,


Answer with any of the scripting language or shell is accepted.

  • Is the prefix always the same length?
    – FelixJN
    Jun 16, 2021 at 7:22
  • @Kusalananda foo and bar are just some example names, the files actually do exist and I wanted only one of them in the directory and the remaining files to be deleted. That one file can be any of the files that match a particular prefix. Jun 16, 2021 at 7:23
  • @FelixJN No, the prefix lengths can vary. I have multiple prefixes with different lengths Jun 16, 2021 at 7:24
  • How about rm *.__?* ? The ? forces at least one character to be there. Of course, that only works, if there is a foo.__.pdf for every foo.
    – Boldewyn
    Jun 17, 2021 at 9:15

6 Answers 6


declare -A seen

for name in *.__*.pdf; do

        if [[ -z ${seen[$prefix]} ]]; then
                printf 'keeping "%s"\n' "$name"
                printf 'deleting "%s"\n' "$name"
                # rm -f -- "$name"

The script above extracts the prefix from each filename matching the filename globbing pattern *.__*.pdf in the current directory. If the prefix has not been seen before, the file is kept. Otherwise the file is deleted (the rm command is currently commented out for safety).

To track what prefixes have been seen, they are stored as keys in an associative array called seen. Associative arrays were introduced in bash release 4.

Since any file matching *.__*.pdf with the same prefix is "equivalent", just renaming all those files to the same name would reduce them down to a single file.

This does not require an associative array and can easily be done by /bin/sh:


for name in *.__*.pdf; do

        printf 'moving "%s" to "%s.__.pdf"\n' "$name" "$prefix"
        # mv -f -- "$name" "$prefix.__.pdf"

Here, all files with the prefix foo are moved to the name foo.__.pdf (the mv command is commented out for safety).


With zsh:

typeset -A hash
for f ($all) hash[${f%%.__*}]=$f
rm -f -- ${all:|keep}

Among the ones with same prefix, that will keep the one that comes last in lexical order. You can reverse that order with all=(*.__*.pdf(On)) instead of all=(*.__*.pdf).


With zsh:

declare -A h=()
rm -- *(e:'((h[${REPLY%%.*}]++))':)

Unfortunately, this doesn't allow setting the order. Adding the o/O glob qualifiers here would only alter the order after excluding the first of each, and e evaluates whether to include/exclude a file in the order the files are found (i.e. unordered as *(oN) would return them).


Using gawk:

ls | awk '/\.__.*.pdf/{a=$0; 
sub(/__.*.pdf/, "",a); 
if (f!=$0) print "rm", $0}'  

For printing files that are desired, the following would do

ls | awk '/\.__.*.pdf/{a=$0; 
sub(/__.*.pdf/, "",a); 
if (f==$0) print ; else print "rm", $0}'

Using perl we can open a directory handle on the current directory and read off all plain files whose names match the specified criterion. The first of those files having the same prefix have their names printed to stdout whilst the subsequent ones with that prefix are deleted. The prefixes are made keys of a hash %h

perl -le '
  opendir my $dh, "."
    or die qq(Cannot opendir ".": $!);

  -f && /^(.+?)\.__.*\.pdf$/ and
    !$h{$1}++ ? print() : unlink()
  while readdir $dh;

  closedir $dh;

Using find command:

fx() {
  set -- "$1" "$tmpdir/${1%%.__*.pdf}"
  if [ ! -s "$2" ]; then
    echo x > "$2"
    shift "$#"
    rm -f "$1"
  return "$#"
export -f fx
tmpdir="$(mktemp -d)" \
find . ! -name . -prune -type f \
-name '?*.__*.pdf' \
-exec sh -c 'fx "$1"' find-sh {} \; -print;

Using python3 along with the pathlib module with the same idea as above:

python3 -c 'from pathlib import Path
seen = {}
for f in Path(".").glob("?*.__*.pdf"):
  if f.is_file():
    p = f.name.find(".__")
    k = f.name[0:p]
    if p > 0:
      if k in seen: f.unlink()

You may use a combination of find and awk:

 find ./ -mindepth 1 -maxdepth 1 -type f -print0 |
 awk 'BEGIN {RS="\x00" ; FS="."}
      seen[$2]++ {system("rm '\''"$0"'\''")}'

This uses NUL as file separator so also can handle things like newlines in names. Assumes that . is used for the prefix separator! If you want the first (by dictionary order) of each batch, you may also pipe find's results through sort:

 find ./ -mindepth 1 -maxdepth 1 -type f -print0 | sort -z |
 awk ....

For exactly matching your pattern .__, you may as well use:

awk 'BEGIN {RS="\x00" ; FS=".__" } seen++[$1] { ... }'
  • But I cannot hardcode "foo" as there can be many files with some other prefix. Jun 16, 2021 at 7:11
  • @AkashTadwai adapted answer after properly understanding your requirements.
    – FelixJN
    Jun 16, 2021 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.