text processing - How to sort unique the output of find that has duplicate filenames from different directory?

Question

I want to sort unique this output of find command without duplicate filenames in any directory.

find /path/to/first_directory/* /path/to/second_directory/* /path/to/third_directory/* -mtime -1 -name "filename_pattern*"

example output:

/path/to/first_directory/sample_file1_2017Dec25.dat
/path/to/first_directory/sample_file2_2017Nov01.dat
/path/to/first_directory/sample_file3_2017Oct08.dat
/path/to/first_directory/archive/sample_file1_2017Dec25.dat.Z
/path/to/first_directory/archive/sample_file2_2017Nov01.dat.Z
/path/to/second_directory/sample_file4_2017Sep11.dat
/path/to/second_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file1_2017Dec25.dat
/path/to/third_directory/sample_file2_2017Nov01.dat
/path/to/third_directory/sample_file3_2017Oct08.dat
/path/to/third_directory/sample_file4_2017Sep11.dat
/path/to/third_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file6_2017July04.dat
/path/to/third_directory/sample_file6_2017June12.dat
/path/to/third_directory/sample_file7_2017May01.dat

From the output you can see that there are duplicate filenames in the /first_directory/ and /first_directory/archive/ and also all files from /first_directory/* and /second_directory/* are also inside /third_directory/*. It means that /third_directory/* is the archive directory of all files found in /first_directory/* and /second_directory/* but there are also files that can only be found in /third_directory/* ( check sample_file6 and sample_file7 )

All I want to print is the files originating from /first_directory/ to /first_directory/archive/ to /second_directory/ to /third_directory/ in this order without there duplicate and also sorted by their date.

Desired output:

/path/to/first_directory/sample_file1_2017Dec25.dat
/path/to/first_directory/sample_file2_2017Nov01.dat
/path/to/first_directory/sample_file3_2017Oct08.dat
/path/to/second_directory/sample_file4_2017Sep11.dat
/path/to/second_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file6_2017July04.dat
/path/to/third_directory/sample_file6_2017June12.dat
/path/to/third_directory/sample_file7_2017May01.dat

Answer 1

If the output of your find command is saved in a file called filelist, then try:

$ awk -F/ '{f=$NF; sub(/\.Z$/,"",f)} !a[f]++' filelist
/path/to/first_directory/sample_file1_2017Dec25.dat
/path/to/first_directory/sample_file2_2017Nov01.dat
/path/to/first_directory/sample_file3_2017Oct08.dat
/path/to/second_directory/sample_file4_2017Sep11.dat
/path/to/second_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file6_2017July04.dat
/path/to/third_directory/sample_file6_2017June12.dat
/path/to/third_directory/sample_file7_2017May01.dat

If you want to do the same thing without creating the file:

find /path/to/first_directory/* /path/to/second_directory/* /path/to/third_directory/* -mtime -1 -name "filename_pattern*" | awk -F/ '{f=$NF; sub(/\.Z$/,"",f)} !a[f]++'

Or, you prefer to spread the command out over multiple lines, use:

find /path/to/first_directory/* /path/to/second_directory/* \
  /path/to/third_directory/* -mtime -1 -name "filename_pattern*" |
    awk -F/ '{f=$NF; sub(/\.Z$/,"",f)} !a[f]++'

where we have added \ to the end of the first line because that is bash's line continuation characters. Because the second line ends with |, it does not require a line continuation character.

How it works

First, it is important the the directories be listed in the find command in your order of priority. I see that you have done that already.

1. -F/

   This tells awk to use / as the field separator. This means that the file name will be the last field, $NF.

2. f=$NF; sub(/\.Z$/,"",f)

   This assigns the file name to variable f and then removes the final .Z from f if present.

3. !a[f]++'

   If f has not been seen before, print this line.

Update 1: Removing other extensions

As per the comments, .Z is not the only extension that needs to be removed. There may be other extensions .dat.edi and .dat.bak that should be replaced with simply .dat. In that case:

awk -F/ '{f=$NF; sub(/\.dat.*/,".dat",f)} !a[f]++' filelist

Update 2: Displaying files sorted by timestamp:

awk -F/ '{f=$NF; sub(/\.dat.*/,".dat",f)} !a[f]++' filelist | xargs -d'\n' -r ls -t