0

I have a directory which contains a huge number of XML files.

They are labelled as filename_date_time_checksum.xml, which means I've got thousands of files which are identical but separated by checksum.

Is there a command line I can run where if filename_date_time_*.xml exists, then retain the last modified version and delete the others?

Example:

uk_3345_20190905_1600_b4ec24da7c59c1d889fb22ad9fad34aca882102e.xml
uk_1552_20190905_1605_1a31fd97541bf300d5bf4c0c4a349e00eee5a8fb.xml
uk_1552_20190905_1605_3d307e3ffbb3259a47a1bc1690c17fd291fe2cb0.xml
uk_1552_20190905_1605_7da5fa3b26cbe04eb01c6308c7b680fb4eb2e463.xml
uk_1552_20190905_1605_b4ec24da7c59c1d889fb22ad9fad34aca882102e.xml
uk_1552_20190905_1605_d01c541fc8db736d223a21a29d9766532140fdb8.xml
uk_1552_20190905_1605_fac6793f2f7e5374157c5d08ee555fcf1bbbf5f2.xml
uk_3345_20190905_1600_1a31fd97541bf300d5bf4c0c4a349e00eee5a8fb.xml
uk_3345_20190905_1600_d01c541fc8db736d223a21a29d9766532140fdb8.xml

The files can be generated at anytime. If the files are generated uk_3345_20190905_1600_d01c541fc8db736d223a21a29d9766532140fdb8.xml on 1st Sept 2019 13:44 & uk_3345_20190905_1600_b4ec24da7c59c1d889fb22ad9fad34aca882102e.xml on 2nd Sept 2019 09:00 I want to only retain the most recent file that was generated. The only attribute of the file I need to use is the modification date.

0
1

This is untested:

# find the *latest* file for each prefix
declare -A mtime name
stat -c "%Y %n" *xml |
while read -r time filename; do
    prefix=${filename%_*}

    if (( $time > ${mtime[$prefix]:-0} )); then
        mtime[$prefix]=$time
        name[$prefix]=$filename
    fi
done

# put the filenames into an associative array for easy lookup
declare -A keep
for filename in "${name[@]}"; do
    keep[$filename]=1
done

# look at teach file to determine its fate
for file in *xml; do
    if [[ -v keep[$file] ]]; then
        echo "# keep $file"
    else
        echo "rm $file"
    fi
done

Or, this pipeline should output the files you want to keep:

paste <( printf "%s\n" *.xml) \
      <( printf "%s\n" *.xml | cut -d _ -f 1-4) \
      <( stat -c '%Y' *.xml) |
sort -k2,2 -k3,3rn |
awk '!seen[$2]++ {print $1}'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.