Linux command-line to find duplicate files and only retain most recent

Question

I have a directory which contains a huge number of XML files.

They are labelled as filename_date_time_checksum.xml, which means I've got thousands of files which are identical but separated by checksum.

Is there a command line I can run where if filename_date_time_*.xml exists, then retain the last modified version and delete the others?

Example:

uk_3345_20190905_1600_b4ec24da7c59c1d889fb22ad9fad34aca882102e.xml
uk_1552_20190905_1605_1a31fd97541bf300d5bf4c0c4a349e00eee5a8fb.xml
uk_1552_20190905_1605_3d307e3ffbb3259a47a1bc1690c17fd291fe2cb0.xml
uk_1552_20190905_1605_7da5fa3b26cbe04eb01c6308c7b680fb4eb2e463.xml
uk_1552_20190905_1605_b4ec24da7c59c1d889fb22ad9fad34aca882102e.xml
uk_1552_20190905_1605_d01c541fc8db736d223a21a29d9766532140fdb8.xml
uk_1552_20190905_1605_fac6793f2f7e5374157c5d08ee555fcf1bbbf5f2.xml
uk_3345_20190905_1600_1a31fd97541bf300d5bf4c0c4a349e00eee5a8fb.xml
uk_3345_20190905_1600_d01c541fc8db736d223a21a29d9766532140fdb8.xml

The files can be generated at anytime. If the files are generated uk_3345_20190905_1600_d01c541fc8db736d223a21a29d9766532140fdb8.xml on 1st Sept 2019 13:44 & uk_3345_20190905_1600_b4ec24da7c59c1d889fb22ad9fad34aca882102e.xml on 2nd Sept 2019 09:00 I want to only retain the most recent file that was generated. The only attribute of the file I need to use is the modification date.

Answer 1

This is untested:

# find the *latest* file for each prefix
declare -A mtime name
stat -c "%Y %n" *xml |
while read -r time filename; do
    prefix=${filename%_*}

    if (( $time > ${mtime[$prefix]:-0} )); then
        mtime[$prefix]=$time
        name[$prefix]=$filename
    fi
done

# put the filenames into an associative array for easy lookup
declare -A keep
for filename in "${name[@]}"; do
    keep[$filename]=1
done

# look at teach file to determine its fate
for file in *xml; do
    if [[ -v keep[$file] ]]; then
        echo "# keep $file"
    else
        echo "rm $file"
    fi
done

---

Or, this pipeline should output the files you want to keep:

paste <( printf "%s\n" *.xml) \
      <( printf "%s\n" *.xml | cut -d _ -f 1-4) \
      <( stat -c '%Y' *.xml) |
sort -k2,2 -k3,3rn |
awk '!seen[$2]++ {print $1}'