1

I want to sort unique this output of find command without duplicate filenames in any directory.

find /path/to/first_directory/* /path/to/second_directory/* /path/to/third_directory/* -mtime -1 -name "filename_pattern*"

example output:

/path/to/first_directory/sample_file1_2017Dec25.dat
/path/to/first_directory/sample_file2_2017Nov01.dat
/path/to/first_directory/sample_file3_2017Oct08.dat
/path/to/first_directory/archive/sample_file1_2017Dec25.dat.Z
/path/to/first_directory/archive/sample_file2_2017Nov01.dat.Z
/path/to/second_directory/sample_file4_2017Sep11.dat
/path/to/second_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file1_2017Dec25.dat
/path/to/third_directory/sample_file2_2017Nov01.dat
/path/to/third_directory/sample_file3_2017Oct08.dat
/path/to/third_directory/sample_file4_2017Sep11.dat
/path/to/third_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file6_2017July04.dat
/path/to/third_directory/sample_file6_2017June12.dat
/path/to/third_directory/sample_file7_2017May01.dat

From the output you can see that there are duplicate filenames in the /first_directory/ and /first_directory/archive/ and also all files from /first_directory/* and /second_directory/* are also inside /third_directory/*. It means that /third_directory/* is the archive directory of all files found in /first_directory/* and /second_directory/* but there are also files that can only be found in /third_directory/* ( check sample_file6 and sample_file7 )

All I want to print is the files originating from /first_directory/ to /first_directory/archive/ to /second_directory/ to /third_directory/ in this order without there duplicate and also sorted by their date.

Desired output:

/path/to/first_directory/sample_file1_2017Dec25.dat
/path/to/first_directory/sample_file2_2017Nov01.dat
/path/to/first_directory/sample_file3_2017Oct08.dat
/path/to/second_directory/sample_file4_2017Sep11.dat
/path/to/second_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file6_2017July04.dat
/path/to/third_directory/sample_file6_2017June12.dat
/path/to/third_directory/sample_file7_2017May01.dat
2
  • sorry that was just a typo. I will correct now – WashichawbachaW Dec 6 '17 at 3:38
  • Very good. I updated the answer to use the updated file list. – John1024 Dec 6 '17 at 7:17
1

If the output of your find command is saved in a file called filelist, then try:

$ awk -F/ '{f=$NF; sub(/\.Z$/,"",f)} !a[f]++' filelist
/path/to/first_directory/sample_file1_2017Dec25.dat
/path/to/first_directory/sample_file2_2017Nov01.dat
/path/to/first_directory/sample_file3_2017Oct08.dat
/path/to/second_directory/sample_file4_2017Sep11.dat
/path/to/second_directory/sample_file5_2017Oct05.dat
/path/to/third_directory/sample_file6_2017July04.dat
/path/to/third_directory/sample_file6_2017June12.dat
/path/to/third_directory/sample_file7_2017May01.dat

If you want to do the same thing without creating the file:

find /path/to/first_directory/* /path/to/second_directory/* /path/to/third_directory/* -mtime -1 -name "filename_pattern*" | awk -F/ '{f=$NF; sub(/\.Z$/,"",f)} !a[f]++'

Or, you prefer to spread the command out over multiple lines, use:

find /path/to/first_directory/* /path/to/second_directory/* \
  /path/to/third_directory/* -mtime -1 -name "filename_pattern*" |
    awk -F/ '{f=$NF; sub(/\.Z$/,"",f)} !a[f]++'

where we have added \ to the end of the first line because that is bash's line continuation characters. Because the second line ends with |, it does not require a line continuation character.

How it works

First, it is important the the directories be listed in the find command in your order of priority. I see that you have done that already.

  1. -F/

    This tells awk to use / as the field separator. This means that the file name will be the last field, $NF.

  2. f=$NF; sub(/\.Z$/,"",f)

    This assigns the file name to variable f and then removes the final .Z from f if present.

  3. !a[f]++'

    If f has not been seen before, print this line.

Update 1: Removing other extensions

As per the comments, .Z is not the only extension that needs to be removed. There may be other extensions .dat.edi and .dat.bak that should be replaced with simply .dat. In that case:

awk -F/ '{f=$NF; sub(/\.dat.*/,".dat",f)} !a[f]++' filelist

Update 2: Displaying files sorted by timestamp:

awk -F/ '{f=$NF; sub(/\.dat.*/,".dat",f)} !a[f]++' filelist | xargs -d'\n' -r ls -t
9
  • Thank you for your answer. Can you add on how to sort the output by date? – WashichawbachaW Dec 6 '17 at 3:32
  • Remove that \ in between /second_directory/` and /third_directory/. I think it's a typo because find returns an error if I copy that. – WashichawbachaW Dec 6 '17 at 5:32
  • @WashichawbachaW Sorry about that trouble. I just added a single-line version of that command to the answer which doesn't have the backslash. (The `` is only used when one spreads the command out over multiple lines which I had done for readability.) – John1024 Dec 6 '17 at 7:03
  • hey bud, thank you very much for your answer. I'm getting the right files now and no duplicate in other directories. But still the date is not being sorted out. Is there any way to do that? Also I found out that there are also files with added extensions after the .dat e.g. .dat.edi, .dat.bak etc. Is there a way to neglect this files if they were a duplicate of a file not having that extension or the .dat only file? – WashichawbachaW Dec 7 '17 at 1:52
  • @WashichawbachaW See the two updates that I just added at the end of the answer. – John1024 Dec 7 '17 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.