The first command you posted can't give the results you show, because they don't have colons at the end; presumably you stripped them. The script you refer to does this to select directory paths, which
ls -R displays with a colon appended, but there is nothing preventing a file name & path from ending with a colon and giving a false positive. This also makes your title misleading; you want to keep most directories and exclude only a few.
Question as asked:
There are several different "flavors" (standards) for regular expressions, most similar but with important differences in details. There are two common in Unix and Unix-origin software, called unimaginatively Basic Regular Expression (BRE) and Extended Regular Expression (ERE). There is an even simpler form used in most shells (and standard find) to match filenames (and case choices) (only ? * and [...]) that isn't even called regexp, just pattern. There is an even more extended form defined by Perl, but usable outside, called Perl Compatible Regular Expression (PCRE). See Why does my regular expression work in X but not in Y?
and http://en.wikipedia.org/wiki/Regular_Expression .
(?! "lookahead" is only in PCRE, while standard
grep does BRE by default or ERE with
-E, although it appears some versions of
grep can do PCRE or you can get and install a separate
pcregrep. But you don't need it. If you wanted non-hidden children of curr dir, just do
'^\./[^.]' depending how strict you want to be. But you say you want no hidden dir anywhere in the path, which is harder to do with a positive regexp, and much easier with negative matching like
grep -v '/\.' .
Backslash is special in both bash (and most if not all shells) and grep (BRE or ERE), so they it must be either doubled
\\ or single-quoted; I prefer the latter. Note double quotes are not sufficient here.
you actually want only directory paths, so as suggested by other answers
find -type d | grep -v /\. is a better approach. That doesn't waste time listing ordinary-file names you then discard. Alternatively you can just use
ls -R | grep :$ without the
-a; by default
ls already skips hidden entries (both ordinary-files and directories). As the script you refer to does!