I have a file with a list of directories in it:


I want to get only /a/b, /a/c, /a/d/e, and /a/e/f/g; that is, I want to exclude lines that have a subset of another line before it. The depth of subdirectories is arbitrary so I could go down 2, 3, 4, etc. directories to find the unique subdirectory.

  • "fixed" at two levels, then, I assume? (What if there was a /a at the top?) – Jeff Schaller Jan 28 at 15:01
  • There's no /a. There may be a /a/c/d and /a/c/d/e but I just want /a/c/d. – mkomarinski Jan 28 at 15:27
  • How do you define what you want? Where do /a/b, /a/c, etc, come from? If you already know what they are, why do you need the file at all? – glenn jackman Jan 28 at 16:09
  • The file was generated - I need to get sizes of the directories in question, but if I'm already getting the size of /a/b then it doesn't make sense to also get /a/b/c, /a/b/d, etc. I just need the set of directories that have no subdirectories. There's the case where I get a /d/e/f/g with no corresponding /d, /d/e, /d/e/f entries. – mkomarinski Jan 29 at 19:25

Assuming your input is sorted, how about checking for a prefix and updating it when it changes?

$ awk 'NR == 1 || ! match($0, "^" pfx) {print; pfx = $0}' file

NOTE: this is a regular expression match, so it may be unsuitable if the entries contain regex special characters - FWIW neither gawk or mawk appear to treat the / as special in this context

  • I can sort the file and this seems to be what I'm looking for. Still waiting on the file output so I'll test again once that's complete and mark it answered once that's done. – mkomarinski Jan 28 at 15:50
gawk -F/ '
        # have we seen something that is a prefix of this line?
        for (prefix in prefixes)
            if ($0 ~ "^" prefix)
                # yes we have

        prefixes[$0] = 1

        # are there prefixes that get "cancelled out" by this new one?
        # e.g. /a/b/c is already a prefix but current line is /a/b
        for (prefix in prefixes)
            if (prefix ~ "^" $0 ".+")
                delete prefixes[prefix]
    END {
        # GNU awk: traverse the array by index, sorted
        PROCINFO["sorted_in"] = "@ind_str_asc"
        for (p in prefixes)
            print p
' list_of_dirs



If you don't have GNU awk, then pipe the output into | sort

  • Your first assumption is incorrect. I don't know the list of directories I'm looking for - they're all embedded in the same file. – mkomarinski Jan 28 at 18:10
  • So, what's your algorithm/criteria for determining which ones you want to keep? – glenn jackman Jan 28 at 18:15
  • Of the set of directories, I want the top-most of each group. So if there's /a/b/c and /a/b, I only want /a/b for my purposes because it would already contain /a/b/c and all the other subdirectories. I don't know how far down I'd have to search otherwise I'd be able to use e.g. find to get what I'm looking for. – mkomarinski Jan 28 at 18:20
  • @mkomarinski, give this effort a try. It makes no assumptions at all. – glenn jackman Jan 28 at 18:38
$ awk -F/ 'NF==3 { print }' filename

We set the field separator to /, and then print lines with only precisely three fields. Presuming your input file format is consistent, only lines such as /a/b will be printed, as the three fields are, in order, an empty string, a, and b.

  • Sorry, should have said that the level of subdirectories is arbitrary. I'll update the question. – mkomarinski Jan 28 at 15:28

Do you have to use awk? Grep can do it. Tyy something like this: egrep '^/a/b\$|/a/c\$'

Depending on your shell, the $ may or may not need to be escaped with the backslash.

  • That presumes pre-knowledge of the answer, though. – Jeff Schaller Jan 28 at 17:10

You can do this using the sed editor as shown:

$ sed -e '
' input_file



  1. Ensure two lines in the pattern space at any time.
  2. If first portion is not found in a leading position in the second portion of the pattern space => they are not of the same branch. We print the first portion, remove it, and go back to reading the next line into the pattern space, and perform the same check.
  3. In the case of a match, we remove the second portion, since that's the larger one (due to sorted input assumption), so we go ahead and promptly remove that portion. And go back and read the next line into the pattern space, and rinse / repeat.

Incase the inputs don't come sorted, you can do it the following way:

$ perl -lne '
    my $l = $_;
    grep !index($l,$_), keys %h or $h{$_}++;
    }{print for sort keys %h;
' input


  • index(str, substr) will return the index where the substr is found within str. To match atthe beginning, a 0 is returned which is then boolean inverted to make it read as success. grep will iterate over all current keys of the hash %h whose keys are the substrings we want.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.