Shell script for moving oldest files?

Question

How do I write a script for moving just the 20 oldest files from one folder to another? Is there a way to grab the oldest files in a folder?

Answer 1

It's easiest in zsh, where you can use the Om glob qualifier to sort matches by date (oldest first) and the [1,20] qualifier to retain only the first 20 matches:

mv -- *(Om[1,20]) target/

Add the D qualifier if you want to include dot files as well. Add . if you want to match only regular files and not directories.

If you don't have zsh, here's a Perl one-liner (you can do it in less than 80 characters, but at a further expense in clarity):

perl -e '@files = sort {-M $b <=> -M $a} glob("*"); foreach (@files[0..1]) {rename $_, "target/$_" or die "$_: $!"}'

With only POSIX tools or even bash or ksh, sorting files by date is a pain. You can do it easily with ls, but parsing the output of ls is problematic, so this only works if the file names contain only printable characters other than newlines.

ls -tr | head -n 20 | while IFS= read -r file; do mv -- "$file" target/; done

Answer 2

Parsing the output of ls is not reliable.

Instead, use find to locate the files and sort to order them by timestamp. For example:

while IFS= read -r -d $'\0' line ; do
    file="${line#* }"
    # do something with $file here
done < <(find . -maxdepth 1 -printf '%T@ %p\0' \
    2>/dev/null | sort -z -n)

What is all this doing?

First, the find commands locates all files and directories in the current directory (.), but not in subdirectories of the current directory (-maxdepth 1), then prints out:

• A timestamp
• A space
• The relative path to the file
• A NULL character

The timestamp is important. The %T@ format specifier for -printf breaks down into T, which indicates "Last modification time" of the file (mtime) and @, which indicates "Seconds since 1970", including fractional seconds.

The space is merely an arbitrary delimiter. The full path to the file is so that we can refer to it later, and the NULL character is a terminator because it is an illegal character in a file name and thus lets us know for sure that we reached the end of the path to the file.

I have included 2>/dev/null so that files which the user does not have permission to access are excluded, but error messages about them being excluded are suppressed.

The result of the find command is a list of all directories in the current directory. The list is piped to sort which is instructed to:

• -z Treat NULL as the line terminator character instead of newline.
• -n Sort numerically

Since seconds-since-1970 always goes up we want the file whose timestamp was the smallest number. The first result from sort will be the line containing the smallest numbered timestamp. All that remains is to extract the file name.

The results of the find, sort pipeline is passed via process substitution to while where it is read as if it were a file on stdin. while in turn invokes read to process the input.

In the context of read we set the IFS variable to nothing, which means that whitespace won't be inappropriately interpreted as a delimiter. read is told -r, which disables escape expansion, and -d $'\0', which makes the end-of-line delimiter NULL, matching the output from our find, sort pipeline.

The first chunk of data, that represents the oldest file path preceded by its timestamp and a space, is read into the variable line. Next, parameter substitution is used with the expression #*, which simply replaces all characters from the beginning of the string up to the first space, including the space, with nothing. This strips off the modification timestamp, leaving only the full path to the file.

At this point the file name is stored in $file and you can do anything you like with it. When you're finished doing something with $file the while statement will loop and the read command will be executed again, extracting the next chunk and the next file name.

Isn't there a simpler way?

No. Simpler ways are buggy.

If you use ls -t and pipe to head or tail (or anything) you'll break on files with newlines in the file names. If you mv $(anything) then files with whitespace in the name will cause breakage. If you mv "$(anything)" then files with trailing newlines in the name will cause breakage. If you read without -d $'\0' then you'll break on files with whitespace in their names.

Perhaps in specific cases you know for sure that a simpler way is sufficient, but you should never write assumptions like that in to scripts if you can avoid doing so.

Solution

#!/usr/bin/env bash

# move to the first argument
dest="$1"

# move from the second argument or .
source="${2-.}"

# move the file count in the third argument or 20
limit="${3-20}"

while IFS= read -r -d $'\0' line ; do
    file="${line#* }"
    echo mv "$file" "$dest"
    let limit-=1
    [[ $limit -le 0 ]] && break
done < <(find "$source" -maxdepth 1 -printf '%T@ %p\0' \
    2>/dev/null | sort -z -n)

Call like:

move-oldest /mnt/backup/ /var/log/foo/ 20

To move the oldest 20 files from /var/log/foo/ to /mnt/backup/.

Note that I am including files and directories. For files only add -type f to the find invocation.

Thanks

Thanks to enzotib and Павел Танков for improvements to this answer.

Answer 3

Combine ls -t output with tail or head.

Simple example, which works only if all file names contain only printable characters other than whitespace and \[*?:

 mv $(ls -1tr | head -20) other_folder

Answer 4

You can use GNU find for this:

find -maxdepth 1 -type f -printf '%T@ %p\n' \
  | sort -k1,1 -g | head -20 | sed 's/^[0-9.]\+ //' \
  | xargs echo mv -t dest_dir

Where find prints the modification time (in seconds from 1970) and the name of each file of the current directory, the output is sorted according to the first field, the 20 oldest are filtered and moved to dest_dir. Remove the echo if you have tested the command line.

Answer 5

No one has (yet) posted a bash example which caters for embedded newline chars (embedded anything) in the the filename, so here's one. It moves the 3 oldest (mdate) regular files

move=3
find . -maxdepth 1 -type f -name '*' \
 -printf "%T@\t%p\0" |sort -znk1 | { 
  while IFS= read -d $'\0' -r file; do
      printf "%s\0" "${file#*$'\t'}"
      ((--move==0)) && break
  done } |xargs -0 mv -t dest

This is the test-data snippet

# make test files with names containing \n, \t and "  "
rm -f '('?[1-4]'  |?)'
for f in $'(\n'{1..4}$'  |\t)' ;do sleep .1; echo >"$f" ;done
touch -d "1970-01-01" $'(\n4  |\t)'
ls -ltr '('?[1-4]'  |'?')'; echo
mkdir -p dest

Here is the check-results snippet

  ls -ltr '('?[1-4]'  |'?')'
  ls -ltr   dest/*

Answer 6

It's easiest to do with GNU find. I use it every day on my Linux DVR to delete recordings from my video surveillance system older than a day.

Here is the syntax:

find /path/to/files/* -mtime +number_of_days -exec mv {} /path/to/folder \;

Remember that find defines a day as 24 hours from the time of execution. Therefore files last modified at 11 pm won't get deleted at 1 am.

You can even combine find with cron, so deletions can be scheduled automatically by running the following command as root:

crontab -e << EOF
@daily /usr/bin/find /path/to/files/* -mtime +number_of_days -exec mv {} /path/to/folder \;
EOF

You can always get more information about find by consulting it's manual page:

man find