I have two long lists of filenames, and I want to merge these two by excluding duplicated filenames in the sense of basename. For example, I have something like


in list A and


in list B. I want to exclude the duplicated file file_1 and file_2 in the merged list. Is there any quick way to do this?

  • 1
    do you mean you do not want to see any occurence of a file whose basename is a duplicate? or just see one occurence (and if so, which one?) – Olivier Dulac Dec 11 '13 at 19:35
  • 1
    Does either list contain any duplicated basenames? i.e., could list B contain both ../data_folderB/file_3 and ../data_folderC/file_3 ? – Mark Plotnick Dec 11 '13 at 19:42

in awk you could do

awk -F "/" ' 
     { filenames[$NF]=$0 ; occurence[$NF]++ ; }
 END { for (basename in filenames) 
       {   if (occurence[basename] == 1) 
           {   print filenames[basename]
     } '  listA  listB

There is of course a terser way to do this, and I abuse "{ ... }", but I hope it's clear:

  • we first fill a "filenames[]" array using index name $NF (= LAST field in the line using "/" as separator, ie the basename).

  • And we also count the number of $NF we saw, thanks to the occurence[] array (if more than "1", we only have the latest one in filenames[$NF], and we have occurence[$NF]>1)

  • Then we only print those that have a occurence == 1


Assuming that you have your lists in some files path/to/list1.txt and /path/to/list2.txt you can use a simple Python program to add the two lists in two Python lists and then set union them as follows:

files1 = [line.strip() for line in open('/path/to/list1.txt')]
files2 = [line.strip() for line in open('/path/to/list2.txt')]

uniqueset = set(files1).union(set(files2))

The uniqueset will contain the unique files. May be you want to exclude the common names altogether from the resulting list. In that case you can do the following:

 uniqueset = set(files1).symmetric_difference(set(files2))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.