Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a list of relative paths such as this:


In the example above, the specifier dir2/file1 (for example) is redundant, because the dir2 entry would include this file.

Want I want to do, essentially, is remove redundant paths from a given list of paths. The above example would output the following:


Note that the files and directories specified need not actually exist on the filesystem.

I am willing to use any common Unix command (sed, awk, perl, etc.).

share|improve this question
If they do not exist on the filesystem, how can you tell the difference between a file and a directory? i.e. how to include dir2/dir3 but not dir2/file1. (Assuming you have not adopted a such a prefix notation for naming every file and directory...) – mr.spuratic Mar 6 '13 at 11:59
@mr.spuratic It doesn't matter at all... For my purposes, the paths may just as well be named x/y and x/z and it would not affect the result – Joshua Spence Mar 6 '13 at 13:02

Let's say the list of relative paths is in file name "path".

You can use the below awk script to print get the desired output:

 $ awk 'BEGIN{FS="/";} {arr[$1]=$0;count[$1]+=1;} END{for(i in arr){if(count[i]==1){print arr[i]}else{print i}}}' path

Hope you understand how it works.

share|improve this answer
Really close. I should've mentioned that it needs to work recursively as well. I will update my example. – Joshua Spence Mar 6 '13 at 5:59
Also, your solution prints out dir3, which it should not – Joshua Spence Mar 6 '13 at 6:01
up vote 0 down vote accepted

Ok, I'm going to answer my own question.

I made my own awk script based on @pradeepchhetri's solution.

cat data.txt | sort | awk '
        if($0 in paths){next;}
END{for(path in paths){print path;}}' | sort

Which works for my purposes.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.