Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a list of relative paths such as this:


In the example above, the specifier dir2/file1 (for example) is redundant, because the dir2 entry would include this file.

Want I want to do, essentially, is remove redundant paths from a given list of paths. The above example would output the following:


Note that the files and directories specified need not actually exist on the filesystem.

I am willing to use any common Unix command (sed, awk, perl, etc.).

share|improve this question
If they do not exist on the filesystem, how can you tell the difference between a file and a directory? i.e. how to include dir2/dir3 but not dir2/file1. (Assuming you have not adopted a such a prefix notation for naming every file and directory...) –  mr.spuratic Mar 6 '13 at 11:59
@mr.spuratic It doesn't matter at all... For my purposes, the paths may just as well be named x/y and x/z and it would not affect the result –  Joshua Spence Mar 6 '13 at 13:02

3 Answers 3

Actually, if you can get the list sorted in the right order (as it already is in your sample), it can be just a matter of checking whether the current file is included in the previous one.

So you could define a new sorting order where / is first. With the GNU libc, you could do it as (zsh or bash or ksh93 syntax below):

mkdir ~/.locales
localedef -i <(
    printf '%s\n' LC_COLLATE 'order_start forward'
    printf '<U%04X>\n' 47 {1..46} {48..255}
    printf '%s\n' UNDEFINED order_end 'END LC_COLLATE'
  ) -f <(zcat /usr/share/i18n/charmaps/ANSI_X3.4-1968.gz
  ) -c ~/.locales/files

You can use that with LOCPATH=~/.locales LC_ALL=files sort.

Then just do:

LOCPATH=~/.locales LC_ALL=files sort < your-list | awk '
  NR > 1 && substr($0, 1, n) == l {next}
  {l = $0 "/"; n = length(l); print}'

Which on your list returns:


You can get away without defining a new sort order if you know that the file names don't contain any character that sorts before / in your locale (in the POSIX/C locale that includes dash space and dots though, so probably not an option). In which case you can just use sort alone.

Or if you know the maximum directory depth in advance, you can use:

sort -t/ -k1,1 -k2,2 -k3,3 -k4,4 -k5,5

For instance for a maximum depth of 5.

share|improve this answer
This is good, but my solution needs to go into a Makefile and so I don't want to be creating a locale on the filesystem for this –  Joshua Spence Mar 6 '13 at 13:03
@JoshuaSpence, see the last part of the answer for conditions where you don't need to create a locale. Note that your sample is already in the right order, so if it's always the case, you may even not need to sort at all. –  Stéphane Chazelas Mar 6 '13 at 14:31

Let's say the list of relative paths is in file name "path".

You can use the below awk script to print get the desired output:

 $ awk 'BEGIN{FS="/";} {arr[$1]=$0;count[$1]+=1;} END{for(i in arr){if(count[i]==1){print arr[i]}else{print i}}}' path

Hope you understand how it works.

share|improve this answer
Really close. I should've mentioned that it needs to work recursively as well. I will update my example. –  Joshua Spence Mar 6 '13 at 5:59
Also, your solution prints out dir3, which it should not –  Joshua Spence Mar 6 '13 at 6:01
up vote 0 down vote accepted

Ok, I'm going to answer my own question.

I made my own awk script based on @pradeepchhetri's solution.

cat data.txt | sort | awk '
        if($0 in paths){next;}
END{for(path in paths){print path;}}' | sort

Which works for my purposes.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.