Keeping the number of files in the folder constant by deleting old files

Question

I'm trying to create a script and run it in crontab every 5 min, so that the number of files in a folder always remain 50000. If there are more, I want the script to delete the old files.

#!/bin/bash
LIMIT=500000
NO=0
#Get the number of files, that has `*.pcap` in its name, with last modified time 5 days     ago

NUMBER=$(find /mnt/md0/capture/DCN/ -maxdepth 1 -name "*.pcap" |wc -l)
if [[ $NUMBER -gt $LIMIT ]]  #if number greater than limit
 then
  del=$(($NUMBER-$LIMIT))
   if [ "$del" -lt "$NO" ]
    then
     del=$(($del*-1))
   fi
   echo $del
   FILES=$(
     find /mnt/md0/capture/DCN/ -maxdepth 1 -type f -name "*.pcap" -print0 |
       xargs -0 ls -lt |
       tail -$del |
       awk '{print $8}'
   )
  rm -f ${FILES[@]}
  #delete the originals

 fi

It doesn't really work, it doesn't run as the number of files are too large. Is there any other method to get this done?

Answer 1

For those not wanting to make assumptions on the names of the files:

With zsh:

#! /bin/zsh -
keep=5000
rm -f /mnt/md0/capture/DCN/*.pcap(D.om[$((keep+1)),-1])

That's using zsh globbing qualifiers:

• D: includes hidden files (Dot files).
• .: only regular files (like find's -type f)
• om: reverse order on the age (based on modification time)
• [$((keep+1)),-1]: only include the 5001^st to the last.

(it may fail if the list of files to remove is very big, in which case you may want to use zargs to split it, or enable zsh's builtin rm with zmodload zsh/files).

With relatively recent versions of GNU tools:

cd /mnt/md0/capture/DCN/ &&
  find . -maxdepth 1 -name '*.pcap' -type f -printf '%T@@%p\0' |
    sort -zrn | sed -z "s/[^@]*@//;1,$keep d" | xargs -r0 rm -f

(assuming GNU sed 4.2.2 or above (2012) for -z, GNU sort 1.14 or above (1996) for -z)

find builds a NUL delimited list of filenames with a Unix timestamp prepended (like 1390682991.0859627500@./file) which is sorted by sort. sed removes the timestamp and prints only from the 5001^st record. That's passed as arguments to rm using xargs -r0.

or (with any version of GNU tools):

cd /mnt/md0/capture/DCN/ &&
  find . -maxdepth 1 -name '*.pcap' -type f -printf '%T@@%p\0' |
    tr '\0\n' '\n\0' | sort -rn | tail -n "+$(($keep+1))" |
    cut -d @ -f2- | tr  '\0\n' '\n\0' | xargs -r0 rm -f

Same, except that we're using cut to remove the timestamp and tail to select the lines starting from 5001. Because GNU cut and tail don't support a -z to work on NUL delimited records, we use tr to swap the newline and NUL characters before and after feeding the data to them.

With GNU ls (4.0 (1998) or above), and bash:

shopt -s dotglob
cd /mnt/md0/capture/DCN/ &&
  eval "files=($(ls -dt --quoting-style=shell-always -- *.pcap))" &&
  rm -f -- "${files[@]:$keep}"

(that also may fail if the list of file is big. Also note that it may include non-regular pcap files (no -type f)).

Standardly/POSIXly/portably, that's a lot trickier:

cd /mnt/md0/capture/DCN/ &&
  ls -dt ./.pcap ./.*.pcap ./*.pcap | awk -v keep="$keep" '
    function process() {
      if (++n > keep) {
        gsub(/[ \t\n"\\'\'']/,"\\\\&", file)
        print file
        file = ""
      }
    }
    /\// {
      if (NR > 1) process()
      file=$0
      next
    }
    {file = file "\n" $0}
    END {if (NR > 0) process()}' | xargs rm -f

(again, you may reach the limit of the number of arguments, and it does not check for regular files).

The tricky bit there is to handle the filenames with newline characters. Above, we're passing ./* to ls which means / will be included once for each file name, and we use that in awk to identify on which line each filename starts, then we know which newline character (in addition to all the other ones special to xargs) to escape for xargs.

Answer 2

I ran the command:

find /mnt/md0/capture/DCN/ -maxdepth 1 -type f -name "*.pcap" -print0 |
  xargs -0 ls -lt | tail -n "$del" | awk '{print $8}'

The problem that I observed was that awk '{print $8}' prints the time, not the file name. awk '{print $9}' would solve that.

Another problems is that xargs may run ls -lt several times which would give you a number of sorted lists of files one after the other, but the whole list would not be sorted.

But, there appear to be other simplifications one could make. You can get the oldest files with:

ls -dt /mnt/md0/capture/DCN/*.pcap | tail -n "$del"

This assumes, as your post seemed to, that the file names have no spaces, tabs, or newline characters in them.

So, the full command for deleting the oldest $del files could be:

ls -dt /mnt/md0/capture/DCN/*.pcap | tail -n "$del" | xargs rm

MORE: If your file names may contain spaces, tabs, backslashes, or quotes in them (but not newlines), use (assuming GNU ls 4.0 (1998) or newer):

ls -dt --quoting-style=shell-always /mnt/md0/capture/DCN/*.pcap |
  tail -n "$del" | xargs rm

Answer 3

Assuming none of the file names contain space, tab, newline, single quote, double quote or backslash characters, this deletes the oldest files above limit

mkdir t && cd t

# 50500 files, 500 to delete
touch {000001..050500}

limit=50000

ls -t|tail -n "+$(($limit + 1))"|xargs rm 

ls|wc -l
50000

tail -n +50001 shows files above limit.

Answer 4

Just use ls with descending sorting -t:

limit=5000
Cnt=0
for line in `ls -t`
do
  if [[ $Cnt -gt $limit ]]
  then
    rm $line
  fi
  Cnt=`expr $Cnt + 1`
done