Find latest versions of multiple files in multiple directories

Question

I am writing an application and one of the prerequisites is to be able to look at multiple directories and find the latest version of each file.

I've had success with ls and find in order to get the latest files, but not the latest version of EACH file if those same files are located in multiple directories. One of the caveats is that I will not necessarily know what the files are called, but will know the names of the directories.

Example: DIR1, DIR2, and DIR3 each contain a version of FileA and FileB. I need the latest versions of FileA and FileB contained in all three (or more) directories.

Anyone have any ideas?

Answer 1

You're on the right track with your choice of tools:

• ls -t is a good way to sort files ordered by time so you can pick off the latest
• find is the right tool to find files matching some pattern in directories and subdirectories

The tricky part of course is that you need some kind of grouping by filename, and pick the latest file in each group. Because of this requirement, I think you need a loop, where you iterate over each target filename to find its latest version.

Assuming the files are in $dir1, $dir2 or $dir3, you could write a function to find the latest version of some pattern like this:

find_latest() {
    pattern=$1
    ls -t "$dir1/$pattern" "$dir2/$pattern" "$dir3/$pattern" | head -n 1
}

Then let's say if you have the patterns access.log, error.log, x*, then you can loop over them like this, for example:

for pattern in access.log error.log 'x*'; do
    latest=$(find_latest 'a*')
    echo $latest
done

If the above assumption is not true, and the files can be in subdirectories of $dir1, $dir2 or $dir3, then you need to use find, it gets a bit more complex:

find_latest() {
    pattern=$1
    find "$dir1" "$dir2" "$dir3" -name "$pattern" -print0 | xargs -0 ls -t | head -n 1
}

There's a small caveat: if a path contains newline characters, this function won't work well because the head -n 1 step will chop off the part of the path after the newline. I cross my fingers that you don't have such paths ;-)

Answer 2

You can do this with pax:

pax -wrtvZs"|.*/||p" ./DIR[123] "$PWD"

So I'll try to break it down by argument:

• -wr - these are write and read, and together they mean pax should copy files rather than archiving. You can forego the copy as well and just create hardlinks instead with -l.

• -t - this resets all file access times to the state they had before pax read them to check their metadata.

• -v - works verbosely.

• -Z - does not compare the mod times of source files until after all possible name substitutions have completed.

  • It is this (and the next thing) that make this so easy. Without this - and likely the problem you'll have elsewhere - is that DIR1/FILEA and DIR2/FILEA are different files even if they share a basename. And so they're never compared without this and...

• -s - substitute and replace portions of a filename with a standard sed regexp.

  • Here I just reduce all portions of every file to their basenames, and so -Z applies to all FILEAs and only the newest is copied into "$PWD".

I used the following test to verify all of this:

for d in DIR3 DIR1 DIR2
do  cd ~; mkdir -p "$d"; cd "$d"
    sleep 90; touch FILEB FILEA
done; cd ~

...which gets the test set. Here are the resulting mod times:

ls -l ./DIR[123]/FILE[AB]                                
-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:28 ./DIR1/FILEA
-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:28 ./DIR1/FILEB
-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:29 ./DIR2/FILEA
-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:29 ./DIR2/FILEB
-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:26 ./DIR3/FILEA
-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:26 ./DIR3/FILEB

And, so, when I run:

pax -wrtvZs"|.*/||p" ./DIR[123] "$PWD"
ls -l ./FILE[AB]

...the output is...

./DIR1/FILEA >> FILEA
/home/mikeserv/FILEA
./DIR1/FILEB >> FILEB
/home/mikeserv/FILEB
./DIR2/FILEA >> FILEA
/home/mikeserv/FILEA
./DIR2/FILEB >> FILEB
/home/mikeserv/FILEB
./DIR3/FILEA >> FILEA
./DIR3/FILEB >> FILEB

-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:29 ./FILEA
-rw-r--r-- 1 mikeserv mikeserv 0 Dec 20 03:29 ./FILEB

You can see it happen. When -s|||p changes a filename the p modifier prints a message to stderr. So we see the DIR1 files get evaluated first - and copied into $PWD, and next the DIR2 files get the same treatment - but the DIR3 files are not copied because $PWD/FILE[AB] are newer than them now.