1

I have a directory with a lot of files. Each file have the same pattern <id>_data_<date>.csv. What i want to do is to delete all files but to keep the newest per <id>.

Example directory:

10020077_data_2017-07-18_001.csv
10020078_data_2017-07-18_001.csv
10020209_data_2019-04-23_001.csv
10020209_data_2019-04-24_001.csv
10020209_data_2019-04-25_001.csv
10020209_data_2019-04-26_001.csv
10020209_data_2019-04-27_001.csv
10020209_data_2019-04-28_001.csv
10020272_data_2019-04-23_001.csv
10020272_data_2019-04-24_001.csv
10020272_data_2019-04-25_001.csv
10020272_data_2019-04-26_001.csv
10020272_data_2019-04-27_001.csv
10020272_data_2019-04-28_001.csv
10020286_data_2019-04-23_001.csv

Expected result:

10020077_data_2017-07-18_001.csv
10020078_data_2017-07-18_001.csv
10020209_data_2019-04-23_001.csv <-- delete
10020209_data_2019-04-24_001.csv <-- delete
10020209_data_2019-04-25_001.csv <-- delete
10020209_data_2019-04-26_001.csv <-- delete 
10020209_data_2019-04-27_001.csv <-- delete
10020209_data_2019-04-28_001.csv
10020272_data_2019-04-23_001.csv <-- delete
10020272_data_2019-04-24_001.csv <-- delete
10020272_data_2019-04-25_001.csv <-- delete
10020272_data_2019-04-26_001.csv <-- delete
10020272_data_2019-04-27_001.csv <-- delete
10020272_data_2019-04-28_001.csv
10020286_data_2019-04-23_001.csv

In this case i can not work with find -mtime because some ids get new files daily and some others only once a month or sometimes only once a year.

My idea was to group the filenames per id and than to keep the last item. How can i solve this with bash?

  • Cannot help you with a working script but I can give a hint. Make sure you get the files ordered by ID and date in reverse order. Means you get the newest first. Keep track of the previous ID. If the ID has not changed then delete the file. If it has then keep the file and store the ID in the previous ID. – Marco Apr 29 at 14:16
  • My idea was to group the filenames per id and than to keep the last item- so how you will know 10020209_data_2019-04-23_001.csv is last file or 10020209_data_2019-04-28_001.csv? maybe you meant keep the last file by newst data in its name for each same Ids? – αғsнιη Apr 29 at 16:51
2

There's no particular need for bash here; a simple sh script could do it, utilizing the positional array twice. The outer loop picks up all of the desired data files (wildcards for the id and date portions); it pulls off the id portion then starts a subshell to loop through all of the files with that id. That subshell then loops through the naturally date-sorted list of those files and deletes all except for the last one, preserving the newest one.

#!/bin/sh

set -- *_data_*.csv
for f in "$@"
do
  id=${f%%_*}
  # a subshell so we don't clobber $@
  (
        set -- "${id}"_data_*.csv
        while [ "$#" -gt 1 ]
        do
          rm -- "$1"
          echo "DELETE: $1"
          shift
        done
  )
done

I added an echo ... DELETE statement so I could demonstrate the results on the filenames you provided:

DELETE: 10020209_data_2019-04-23_001.csv
DELETE: 10020209_data_2019-04-24_001.csv
DELETE: 10020209_data_2019-04-25_001.csv
DELETE: 10020209_data_2019-04-26_001.csv
DELETE: 10020209_data_2019-04-27_001.csv
DELETE: 10020272_data_2019-04-23_001.csv
DELETE: 10020272_data_2019-04-24_001.csv
DELETE: 10020272_data_2019-04-25_001.csv
DELETE: 10020272_data_2019-04-26_001.csv
DELETE: 10020272_data_2019-04-27_001.csv
0

You can do it with a sequence of commands in a one-liner too, as long as you have mktemp, tee, sort, grep, xargs, and of course rm available on your system. If you don't have tac you can replace it with sort -r:

(temp_all=$(mktemp) && temp_last=$(mktemp) && { tac | tee $temp_all | sort -un > $temp_last ; } && grep -vf $temp_last $temp_all ; rm -f $temp_last $temp_all)

The above expects the whole list of files in stdin (it can come from whatever means you see fit, a find, an ls, a file, etc.) and shows the list of files to delete. You can then pipe such list to an xargs rm

Broken down:

(
temp_all=$(mktemp) && \
temp_last=$(mktemp) && \ # make a couple of temp files
{ 
    tac | \              # reverse the list of files and ...
    tee $temp_all | \    # pipe it into one temp entirely and also ...
    sort -un > $temp_last ; \ # into a sort that makes names unique into the other temp
} && \
    grep -vFf $temp_last $temp_all ; \ # use grep to filter out names
rm -f $temp_last $temp_all  # remove temp files
)

This can deal with any number of input names, but requires that no name has embedded newlines. It seems reasonable for your case.

0
Put all file names in l.txt
Proceed with below steps and it worked fine

da=`awk -F "_" '{print $3}' l.txt | sort | uniq| sort -nr| sed -n '1p'`


 for id in `awk -F "_" '{print $3}' l.txt | sort | uniq`
> do
> find  path -maxdepth 1 -type f -newermt $da -iname "$id*"  | sed -n '2,$p'| awk '{print "rm" " " $1}'| sh;done
0

I know there are already a lot of answers, but here as alternative in Python. You don't need to loop through the files twice.

#!/usr/bin/env python                                                           

import os
import glob

if __name__ == '__main__':
    newest_dict = dict()

    for f in glob.glob('*.csv'):
        id = f[:8]

        if id not in newest_dict:
            newest_dict[id] = f
        else:
            nf = newest_dict[id]

            f_ts = f[14:24]
            nf_ts = nf[14:24]

            if f_ts > nf_ts:
                newest_dict[id] = f
                print("Deleting", nf)
                os.remove(nf)                                                  
            else:
                print("Deleting", f)
                os.remove(f)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.