2

I want to delete old files and keep only the most recent from each file pattern based on the first five characters (aa_te, bb_co, cc_co, dd_xy) and the date and time in the file name.

Note:

All files have .err.mm_dd_yyyy hr_mm_ss am/pm.csv for the sorting. Filename length may vary

Example: I have the below CSV files in my folder c/test

aa_te_bbc_207_mc_ghghgh.err.05_31_2021 7_15_04 am.csv
aa_te_ccd.err.05_30_2021 7_55_35 pm.csv
bb_co_ddee_354_m.err.06_15_2021 5_20_00 am.csv
bb_co_eeffff.err.06_15_2021 5_20_01 am.csv
cc_co_eeffyy_cc.err.04_10-2020 12_00_00 am.csv
dd_xy_eeffyyvv_cc.err.04_10-2020 12_00_00 am.csv
dd_xy_eeffyyvv_cc.err.04_10-2020 12_10_25 am.csv

What I expect remains of those afterwards:

aa_te_bbc_207_mc_ghghgh.err.05_31_2021 7_15_04 am.csv
bb_co_eeffff.err.06_15_2021 5_20_01 am.csv
cc_co_eeffyy_cc.err.04_10-2020 12_00_00 am.csv
dd_xy_eeffyyvv_cc.err.04_10-2020 12_10_25 am.csv
5
  • This would be a lot easier if you used YYYY-mm-dd date format, and 2-digit hours Nov 16 at 21:10
  • I changed the file directory dir= c/test and, removed the echo rm and kept && " $search" at last and run the code. There is no error message but the files are not removed at all. Am I missing something? Was it working for you? Will this code work for hundreds of files with out any change?
    – Promise
    Nov 19 at 2:49
  • Is this a comment on @alecxs's answer? Nov 19 at 3:45
  • never use code you don't fully understand
    – alecxs
    Nov 19 at 16:51
  • i.stack.imgur.com/zDvEj.jpg
    – alecxs
    Nov 22 at 10:11
2

delete "echo" rm in last line when okay for you

#!/bin/bash

# search dir
dir=./

# function conv PATTERN
conv() {
  # extract date from file name: 06_15_2021 5_20_00 am
  date=${1%.*}
  date=${date##*.}

  # sanitize date string: 06/15/2021 5:20:00-am
  date=${date//[^0-9AaPpMm]/-}
  date=${date/-//}
  date=${date/-//}
  date=${date/-/ }
  date=${date/-/:}
  date=${date/-/:}

  # print epoch time
  date --date="$date" +%s || return 1
  return 0
}

# search: *m.csv
for file in "$dir"*[Mm].[Cc][Ss][Vv]
  do
    [ -f "$file" ] || continue
    # file name: bb_co_ddee_354_m.err.06_15_2021 5_20_00 am.csv
    file=${file##*/}
    # base name: bb_co_ddee_354_m
    name=${file%%.*}
    # suffix: ddee_354_m
    del=${name#*_}
    del=${del#*_}
    # short name = base name - _suffix: bb_co
    name=${name%_$del}
    # convert date
    date=$(conv "$file") || continue

    # search short name: bb_co_*m.csv
    for search in "${dir}${name}"_*[Mm].[Cc][Ss][Vv]
      do
        # delete older file
        date2=$(conv "${search##*/}") || continue
        [ "$date" -gt "$date2" ] && echo rm "$search"
    done
done
1
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – terdon
    Nov 22 at 10:33
1

With zsh instead of bash, you could do:

#! /bin/zsh -

zmodload zsh/datetime
typeset -A latest=()
all=(
  *.err.*.csv(Nnoe['LC_ALL=C strftime -rs REPLY %m_%d_%Y_%I_%M_%S_%p ${${${REPLY:r}##*.err.}//[- ]/_}'])
)
for file in $all; do
  latest[$file[1,5]]=$file
done

keep=($latest)
echo rm -f -- ${all:|keep}

(to be run within the directory where the files are; remove echo (for dry-run) if that looks alright)

We sort the list of files by timestamp into $all using zsh's oe glob qualifier which sorts (numerically (with n)) based on the value of $REPLY as returned by the code in [...].

That code uses strftime -r (strftime in reverse which in effect is strptime) to convert the time stamp in the file name to the epoch time in seconds.

Then we loop on that list to record the latest file for each 5-character prefix in to the $latest associative array.

Then we use the ${a:|b} array subtraction operator to remove the elements of $all that are not in $keep (itself containing the values of the $latest associative array).

That assumes all the *.err.*.csv files in the current directory are in the format you specified, though it will also accommodate filename where date elements are separated with -s instead of _s as seen in some of your samples (by converting the -s to _s using the ksh93-style ${var//pattern/replacement} operator).

2
  • @alecxs, oh. I hadn't seen that. But then again, it contradicts the OP's specification. Nov 22 at 10:14
  • 1
    @alecxs, I've updated the code so it converts the -s to _s prior to calling strftime -r in case it wasn't just typos in the OP's sample. Nov 22 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.