7

I want to put a script in cronjob which will run in a particular time and if the file count is more than 60, it will delete oldest files from that folder. Last In First Out. I have tried,

#!/bin/ksh  
for dir in /home/DABA_BACKUP  
do  
    cd $dir  
    count_files=`ls -lrt | wc -l`   
    if [ $count_files -gt 60 ];  
    then  
        todelete=$(($count_files-60))  
        for part in `ls -1rt`  
        do  
            if [ $todelete -gt 0 ]  
            then  
                rm -rf $part  
                todelete=$(($todelete-1))  
            fi  
        done  
    fi
done   

These are all backup files which are saved daily and named backup_$date. Is this ok?

  • 1
    Notes: To just count files you don't need ls options -lrt and to build a list in the for loop you don't need ls option -1. Free variable expansions ("$dir" and "$part") should be quoted. Instead of The backtics use $(ls | wc -l). – Janis Jun 14 '15 at 11:54
  • @Janis that will still fail if the file names contain newlines. – terdon Jun 14 '15 at 11:55
  • 1
    @Yes, I know. There's just too much worth fixing there. – Janis Jun 14 '15 at 11:57
  • My script is ok... I just edited according to the last answer. It's now deleting files from the folder where file count is greater then 60. Last file entered and the first one removed from folder. That what I wanted, Last In First Out. – pmaipmui Jun 14 '15 at 12:01
  • It's not OK. It will break if your file names contain spaces or newlines. It is also far more complex than necessary. What format are your names in? You said backup_$date but what is $date? Is it 114-06-2015? Or Sun Jun 14 15:06:53 EEST 2015? If you tell us exactly what it is, we can give you more robust and efficient way of doing this. – terdon Jun 14 '15 at 12:07
3

No, for one thing it will break on filenames containing newlines. It is also more complex than necessary and has all the dangers of parsing ls.

A better version would be (using GNU tools):

#!/bin/ksh  
for dir in /home/DABA_BACKUP/*
do
    ## Get the file names and sort them by their
    ## modification time
    files=( "$dir"/* );
    ## Are there more than 60?
    extras=$(( ${#files[@]} - 60 ))
    if [ "$extras" -gt 0 ]
    then
    ## If there are more than 60, remove the first
    ## files until only 60 are left. We use ls to sort
    ## by modification date and get the inodes only and
    ## pass the inodes to GNU find which deletes them
    find dir1/ -maxdepth 1 \( -inum 0 $(\ls -1iqtr dir1/ | grep -o '^ *[0-9]*' | 
        head -n "$extras" | sed 's/^/-o -inum /;' ) \) -delete
    fi
done

Note that this assumes that all files are on the same filesystem and can give unexpected results (such as deleting wrong files) if they are not. It also won't work well if there are multiple hardlinks pointing to the same inode.

  • Many Thanks @terdon. I have just modified my script as per your solution. It's working smoothly. Thanks everyone for you valuable efforts. Could you please do me a favor? If possible, then kindly share some links for writing shell scripts. – pmaipmui Jun 14 '15 at 12:20
  • 1
    I can hardly believe that the date format Nainita mentioned (130615, 140615) is automatically sorted well as you assume... Try with dates 140615 and 130715. The default output will be 130715 followed by 140615. – Lambert Jun 14 '15 at 18:22
  • 1
    @mikeserv you raise two valid points. Was the snarky sarcasm really needed to make them? Why must you turn everything into a fight? All you had to do is point out my mistakes and I would have happily admitted them yet you chose to attack instead of teach. – terdon Jun 15 '15 at 11:56
  • 1
    @mikeserv you were snarky now and it was uncalled for. Surely by now you know I have absolutely no problem admitting I was wrong. And I was very wrong here. All you had to do was point it out. Anyway, see updated answer, you'll like it, it parses ls. – terdon Jun 15 '15 at 19:03
  • 1
    @mikeserv I'm not using -l. I also have no idea why you mention Solaris. I'm as familiar with your opinion of that post as you must be with mine. Let's not rehash it. I am finding because that's the best way I know of to delete files by inodes. I'd be happy to hear of a better one (and that would make an actually constructive comment). And yes, this is not a good answer and I'd rather not see it accepted (and I wrote this before seeing your last comment). Since it is accepted, however, I have at least tried to make it i) work, unlike the previous version and ii) robust. – terdon Jun 15 '15 at 22:12
3
#! /bin/zsh -
for dir (/home/DABA_BACKUP/*) rm -f $dir/*(Nom[61,-1])

For the zsh-ignorant ;-):

  • for var (list) cmd: short version of the for var in list; do cmd; done loop (reminiscent of perl syntax).
  • $dir: zsh variables don't need quoted like they do in other shells as zsh has explicit split and glob operators so doesn't do implicit split+glob upon parameter expansion.
  • *(...): glob with glob qualifiers:
  • N: nullglob: the glob expands to nothing instead of raising an error when it doesn't match.
  • m: order the generated files on modification time (youngest first).
  • [61,-1]: from that ordered list pick the 61st to last ones.

So basically removes all but the 60 youngest files.

  • Could you explain that to the zsh ignorant? I assume you are somehow sorting by date so you won't have the issues my answer does, right? Is that what the NOm does? – terdon Jun 15 '15 at 17:19
  • @terdon, see edit. I actually had the logic wrong (reversed). Should be om to sort with youngest first (like in ls -t). – Stéphane Chazelas Jun 15 '15 at 18:58
  • Very nice, thanks! Could you have a look at my updated answer. I think it should i) work now and ii) be robust with any file name. I'd appreciate it if you could point out any file names that would break it. – terdon Jun 15 '15 at 19:06
1

To obtain a list of the oldest entries to delete (thus keeping the 60 latest entries):

ls -t | tail -n +61

Note that the principle problem of your approach remains to be addressed here as well: how to handle files with newlines, in case it matters; otherwise you can just use (replacing your quite complex program):

cd /home/DABA_BACKUP || exit 1
ls -t | tail -n +61 | xargs rm -rf


Note: As it seems that you have daily backups you could maybe also use an approach based on the file dates and find; as in:

find /home/DABA_BACKUP -mtime +60 -exec ls {} +

(where the ls command would - after careful examination of the correct operation! - be replaced by the appropriate rm command).

  • 1
    Note that using xargs also assumes filename don't contain space, tabs, newline (other forms of blank characters depending on the locale and xargs implementation), single quote, double quote and backslash. You may want to add a -- to the rm cmdline to avoid problems with files whose name starts with -. (probably not a problem to the OP but worth noting here for anyone coming here with a similar need). – Stéphane Chazelas Jun 15 '15 at 20:24
1
rm60()( IFS=/; set -f; set $(
        set +f; \ls -1drt ./*)
        while shift &&
              [ $# -gt 60 ]
        do    [ -d "${1%?.}" ] ||
              rm "./${1%?.}"   || exit
        done
)

This will work for you. It will delete the oldest files in the current directory up to a count of 60. It will do this by parsing ls robustly and it will do it without making any assumptions about your filenames - they might be named anything and need not be named by dates. This will only work for a listing of the current directory and in the case that you have a POSIX ls installed (and not masked by some evil shell function, but aliases are covered).

The above solution just applies some very basic shell splitting to some very basic Unix pathnames. It ensures ls lists all not-dot files in the current directory one per line like:

./oldestfile
./second-oldestfile

Now, any one of those might have newlines in between as well, but that would not be an issue. Because in that case they would be listed like:

./oldest
file
./s
econd

old
est
file
./third

...and so on. And the newlines don't bother us anyway - because we don't split on them. Why would we? We're working w/ pathnames, we should split on the path delimiter, and so that is what we do: IFS=/.

Now that works out a little bit strange. We end up with an argument list that looks like this:

<.> <file1\n.> <file2\n.> ... <filelast>

...but that's actually very good for us, because we can delay our arguments being treated by the shell as files (or, in the case we want to avoid, symlinks) until we're quite ready to rm them.

So once we've got our file list all we have to do is shift away our first argument, check to see that we currently have more than 60 arguments, probably decline to rm a child directory (though, of course, that's completely up to you), and otherwise rm our first argument less its last two characters. We don't have to worry about the last last argument - which hasn't got the appended period - because we never get there, and instead quit at 60. If we've made it this far for an iteration then we just try again and loop over the arg list in this fashion until we have pruned it to our satisfaction.

How does this break? It doesn't, to my knowledge, but I've allowed for it - if at any time an unexpected error occurs the loop breaks and the function returns other than 0.

And so ls can do your listing for you in the current directory without any issue at all. You can robustly allow it to sort your arguments for you, as long as you can reliably delimit them. It is for that reason that this will not work as written for anything but the current directory - more than one delimiter in a pathstring would require another level of delimiting, which could be done by factoring it out doubly for all but the last into NUL fields, but I don't care to do that now.

-1

If you know, files are all named backup_*, you should include that in the ls command, so you only handle those and not files that accidentally lands in the directory. Then ls is used in a pipe, it lists only 1 file per line, and then only counting, no need to sort, so

count_files=$(ls -U backup_* | wc -l)

and

for part in $(ls -rt backup_*);do
    rm -rf "$part"
    todelete=$(($todelete-1))
    if [[ $todelete -eq 0 ]]; then
        break
    fi
done
  • 1
    It's generally advised to avoid parsing of ls output in scripts. You could use find instead. – Erathiel Jun 14 '15 at 12:44
  • @Erathiel - exactly what does find offer here that should be preferred to ls? One time somebody wrote a pretty error-strewn blog post about parsing ls and for some reason the entire linux community treat it like the Pentateuch. Look, the few valid points made in the blog post apply equally as well to find in this case. – mikeserv Jun 15 '15 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.