What is the bash equivalent of the where cmdlet?

Question

The where cmdlet is something I use in PS quite often- its very convenient.

If I want to get all files with a name starting with "test" I would do this:

ls | ?{$_.name -like 'test*'}

What is an equally terse way a nix admin would do this in bash? Bash is text not object based obviously so the command would probably be something like:

ls | ?{<column#> -like 'test*'}

I'm not asking about the ls command (ls test* would work, but that's command specific) I'm looking for a terse bash equivalent of how to pipe output through filters like this.

Answer 1

TL;DR: if you don't like that ls stuff even if that is NOT command-specific (explained below), use arr=(); for foo in * [!*]*; do [[ $foo == test* ]] && arr+=("$foo"); done. This is basically a arr=list_files.filter((i) -> (i.glob_match('test*')); as you may write in some other languages.

But if you read on, you will still find out that ls is nice and those testing should be skipped by a direct array assignment like arr=(test*) (mikeserv made me realize I really need to say this first).

A quick review on text filtering

Just like others mentions in the comments, Unix shell simply pipes text streams, not object packs. And since the Unix world allows a lot more things in filenames and in a lot of other spaces than Windows, only shell word boundaries is safe. For things like filenames where \0 is luckily not allowed, you can use it in streams too.

grep and friends keeps serving as the basic text filter. It eats in lines from stdin or files, matches against a given regex and gives out the matching (or the reverse, -v) lines or parts (-o) as stdout, yet another stream which is still unsafe to use.

This works for newline-separated text like most code and poems, but not for filenames. Filenames can contain newlines and grep may see two separate lines for one file from ls.

What globbing looks like to programs in Unix shells

In UNIX shells, wildcards are processed by the shell rather than processed by the target program. This makes things a bit more consistent, and also causes the confusion that makes you think the test* part has something to do with ls.

Suppose we have the files test1, test2, test3, ls only feels like it was called like this:

ls test1 test2 test3

It basically knows nothing about what you did to it.

Internally, the test* is expanded to shell words. And since we have the construct for varname in [word-list]; do [commands]; done, we can make things like this:

for i in *; do
    if i matches the pattern; then
         do something
    fi # that marks endif
done

And there is one thing in bash, [[, that performs pattern matching. For a given constuct [[ lhs == rhs ]], bash checks if lhs matches the pattern on rhs. In our case, we can use [[ $i == test* ]] for that i matches the pattern part. This is not available in a minimal POSIX shell, use case for that.

And we need to add some actions to do something. In bash, there are arrays:

# Arrays doesn't exist in POSIX standard too.
a=() # an empty array, since bash is weak-typed this doesn't mean anything
b=(foo bar baz) # array are assigned with, well, list-of-words.
for i in *; do
    if [ $i == test* ]]; then
         a+=("$i") # quoting avoids some word-splitting and you know what += is
    fi # that marks endif
done
# and here do something to your lonely array, like those described in 
# gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html

One more thing here. In Unix systems by convention filenames starting with . marks a hidden file and * won't include them. .* adds them explicitly, but . means current dir and .. means basedir (same as windows) so you want to get rid of them. Using .[!.]* works, as it means a dot, a character that is not a dot and any number including zero of characters. This has nothing to do with your test* thing, since test does not start with ..

And for multiple pattern 'or' we can use the extglob (um, run shopt -s extglob first) thing @(patt1|patt2), or we can wrap the whole thing with a case where multiple patterns are allowed, ...

But why do you need such thing? Just use a=(test*). test* gives you a list of words if there is a match. For multiple patterns, use a=(test* tset*).

But what if glob matching failed?

If a glob matches nothing, it remains as the glob pattern itself in the word list. That looks good for lazy shells users, but not for serious scripters like us.

Luckily there is a shopt in bash called nullglob, which doesn't put the glob pattern back. Just use shopt -s nullglob to enable it first. If you want it portable among other POSIX shells, well, let's go back to for-and-case-filtering.

How do people get to that ls solution

In the following -- is added in case what you are globbing is not test* but *test*, and you may get some filename like -test123 and be recognized as some ls options. -- marks end-of-options by convention.

# So what you need is globbing:
echo test*
# But those are not clear enough since echo only adds spaces between them. Use newlines:
printf '%s\n' test*
# But our screen doesn't have so many lines. Ah, yes, ls can make it into columns:
ls -- test*
# But it lists the contents of the directory test233/. Let's ask it not to:
ls -d -- test*
# Oh, good enough, let's add some color and some pretty type indicator:
ls -dF --color=auto -- test*

So for your simple question it goes like: ls -d test*. But it is NOT for a single command; we are using ls only for pretty-printing.

So is filenames in pipes always stupid?

No. Some programs tried to keep people think they're safe to use by adding/using the \0 deliminator, like find -print0 and xargs -0. Unluckily it requires some hacks to let the shells accept \0, so..

And there is still a better globbing solution in many shells. find walks a directory recursively and conditionally prints the found filenames, and it is often used a way to list files recursively. In bash, we have this:

shopt -s globstar
for i in **; do
    try-some-test || continue
    do-something
done

Which does the work just well.

But bash doesn't provide everything, e.g. it doesn't have neat parallel commands and it's never as fast as native code running. That's why people use other programs like parallel do things. Since we have \0, well, it's not that bad, and we can use find -print0 to feed it.

Answer 2

This will return only the files that start with "test":

ls | grep '^test'

More examples:

That will return only lines which contain the word "test". I am using ls -al" instead of "ls" alone to have a 1 line per record result:

ls -al | grep "test"

This would return any line containing test or word2:

ls -al | grep "test\|word2"

This would return any line not containing the string "test":

ls -al | grep -v "test"

If you really want to go by column, you can do like that:

ls -al | awk '$9 ~ /^test/'

Where $9 would mean column #9 which refers to the filename on my Debian machine, and the following regexp says that it starts with "test". awk separates columns by any run of spaces and/or tabs and/or newlines is treated as a field separator.

Answer 3

If you want to filter out all files beginning (or more complex) with "test", you can use egrep to apply regex pattern matching.

Searching ls output for files beginning with the letter D

$ ls |egrep "^D"
Desktop
Documents
Downloads
Dropbox

If you don't uses regex, you get this

$ ls |grep D
Desktop
Documents
Downloads
Dropbox
README
Vuze Downloads