1

The where cmdlet is something I use in PS quite often- its very convenient.

If I want to get all files with a name starting with "test" I would do this:

ls | ?{$_.name -like 'test*'}

What is an equally terse way a nix admin would do this in bash? Bash is text not object based obviously so the command would probably be something like:

ls | ?{<column#> -like 'test*'}

I'm not asking about the ls command (ls test* would work, but that's command specific) I'm looking for a terse bash equivalent of how to pipe output through filters like this.

2
  • You're looking for a Linux shell tutorial, and I close-voted as off-topic because of it. But for particularly what you asked, awk/gawk will separate lines into fields and allow you to match specific columns, e.g. ls -lha | awk '$9~/test/' will print the full ls -lha output if field 9 matches "test" (field 9 is the filename here, but I don't know it always will be). See also cut if the fields are neatly delimited, and the below answers with grep.
    – TessellatingHeckler
    Oct 23 '15 at 22:24
  • As you note, bash is text-based, not object-based. So, processing filenames using pipes, etc. without precautions can be a bad idea. See mywiki.wooledge.org/ParsingLs, unix.stackexchange.com/questions/128985/why-not-parse-ls
    – muru
    Oct 24 '15 at 7:21
1

TL;DR: if you don't like that ls stuff even if that is NOT command-specific (explained below), use arr=(); for foo in * [!*]*; do [[ $foo == test* ]] && arr+=("$foo"); done. This is basically a arr=list_files.filter((i) -> (i.glob_match('test*')); as you may write in some other languages.

But if you read on, you will still find out that ls is nice and those testing should be skipped by a direct array assignment like arr=(test*) (mikeserv made me realize I really need to say this first).

A quick review on text filtering

Just like others mentions in the comments, Unix shell simply pipes text streams, not object packs. And since the Unix world allows a lot more things in filenames and in a lot of other spaces than Windows, only shell word boundaries is safe. For things like filenames where \0 is luckily not allowed, you can use it in streams too.

grep and friends keeps serving as the basic text filter. It eats in lines from stdin or files, matches against a given regex and gives out the matching (or the reverse, -v) lines or parts (-o) as stdout, yet another stream which is still unsafe to use.

This works for newline-separated text like most code and poems, but not for filenames. Filenames can contain newlines and grep may see two separate lines for one file from ls.

What globbing looks like to programs in Unix shells

In UNIX shells, wildcards are processed by the shell rather than processed by the target program. This makes things a bit more consistent, and also causes the confusion that makes you think the test* part has something to do with ls.

Suppose we have the files test1, test2, test3, ls only feels like it was called like this:

ls test1 test2 test3

It basically knows nothing about what you did to it.

Internally, the test* is expanded to shell words. And since we have the construct for varname in [word-list]; do [commands]; done, we can make things like this:

for i in *; do
    if i matches the pattern; then
         do something
    fi # that marks endif
done

And there is one thing in bash, [[, that performs pattern matching. For a given constuct [[ lhs == rhs ]], bash checks if lhs matches the pattern on rhs. In our case, we can use [[ $i == test* ]] for that i matches the pattern part. This is not available in a minimal POSIX shell, use case for that.

And we need to add some actions to do something. In bash, there are arrays:

# Arrays doesn't exist in POSIX standard too.
a=() # an empty array, since bash is weak-typed this doesn't mean anything
b=(foo bar baz) # array are assigned with, well, list-of-words.
for i in *; do
    if [ $i == test* ]]; then
         a+=("$i") # quoting avoids some word-splitting and you know what += is
    fi # that marks endif
done
# and here do something to your lonely array, like those described in 
# gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html

One more thing here. In Unix systems by convention filenames starting with . marks a hidden file and * won't include them. .* adds them explicitly, but . means current dir and .. means basedir (same as windows) so you want to get rid of them. Using .[!.]* works, as it means a dot, a character that is not a dot and any number including zero of characters. This has nothing to do with your test* thing, since test does not start with ..

And for multiple pattern 'or' we can use the extglob (um, run shopt -s extglob first) thing @(patt1|patt2), or we can wrap the whole thing with a case where multiple patterns are allowed, ...

But why do you need such thing? Just use a=(test*). test* gives you a list of words if there is a match. For multiple patterns, use a=(test* tset*).

But what if glob matching failed?

If a glob matches nothing, it remains as the glob pattern itself in the word list. That looks good for lazy shells users, but not for serious scripters like us.

Luckily there is a shopt in bash called nullglob, which doesn't put the glob pattern back. Just use shopt -s nullglob to enable it first. If you want it portable among other POSIX shells, well, let's go back to for-and-case-filtering.

How do people get to that ls solution

In the following -- is added in case what you are globbing is not test* but *test*, and you may get some filename like -test123 and be recognized as some ls options. -- marks end-of-options by convention.

# So what you need is globbing:
echo test*
# But those are not clear enough since echo only adds spaces between them. Use newlines:
printf '%s\n' test*
# But our screen doesn't have so many lines. Ah, yes, ls can make it into columns:
ls -- test*
# But it lists the contents of the directory test233/. Let's ask it not to:
ls -d -- test*
# Oh, good enough, let's add some color and some pretty type indicator:
ls -dF --color=auto -- test*

So for your simple question it goes like: ls -d test*. But it is NOT for a single command; we are using ls only for pretty-printing.

So is filenames in pipes always stupid?

No. Some programs tried to keep people think they're safe to use by adding/using the \0 deliminator, like find -print0 and xargs -0. Unluckily it requires some hacks to let the shells accept \0, so..

And there is still a better globbing solution in many shells. find walks a directory recursively and conditionally prints the found filenames, and it is often used a way to list files recursively. In bash, we have this:

shopt -s globstar
for i in **; do
    try-some-test || continue
    do-something
done

Which does the work just well.

But bash doesn't provide everything, e.g. it doesn't have neat parallel commands and it's never as fast as native code running. That's why people use other programs like parallel do things. Since we have \0, well, it's not that bad, and we can use find -print0 to feed it.

10
  • @mikeserv Because he wants explicit filtering like (filter (lambda (i) (glob-match i "test*")) (list-files)). Oct 24 '15 at 17:46
  • @mikeserv But he wants to see it in action. Oct 24 '15 at 17:50
  • well, from what i understand, the asker wants to reliably prune piped input based on shell glob patterns. you dont do that here - you filter an array into another array in the same shell process. i dont think it fits.
    – mikeserv
    Oct 24 '15 at 17:53
  • @mikeserv glob patterns still require good deliminator(s), which is almost unavailable in shell. Oct 24 '15 at 18:08
  • 1
    nevermind, man. i still dont think it answers the question and instead attempts to avoid it.
    – mikeserv
    Oct 24 '15 at 18:47
0

This will return only the files that start with "test":

ls | grep '^test'

More examples:

That will return only lines which contain the word "test". I am using ls -al" instead of "ls" alone to have a 1 line per record result:

ls -al | grep "test"

This would return any line containing test or word2:

ls -al | grep "test\|word2"

This would return any line not containing the string "test":

ls -al | grep -v "test"

If you really want to go by column, you can do like that:

ls -al | awk '$9 ~ /^test/'

Where $9 would mean column #9 which refers to the filename on my Debian machine, and the following regexp says that it starts with "test". awk separates columns by any run of spaces and/or tabs and/or newlines is treated as a field separator.

1
  • You don't need ls -l (ell) to get one filename per line when piping, or redirecting; the multi-column format is used only when outputting to a tty, and even there it can also be turned off by -1 (one). -a (with a directory operand, or the default of .) increases the set of files listed/searched; this may or may not be wanted Oct 26 '15 at 9:42
0

If you want to filter out all files beginning (or more complex) with "test", you can use egrep to apply regex pattern matching.

Searching ls output for files beginning with the letter D

$ ls |egrep "^D"
Desktop
Documents
Downloads
Dropbox

If you don't uses regex, you get this

$ ls |grep D
Desktop
Documents
Downloads
Dropbox
README
Vuze Downloads
5
  • What's wrong with straightforward globbing: ls -d D* for the first example and ls -d *d* for the second?
    – roaima
    Oct 24 '15 at 22:11
  • @roaima The OP said he knew about that and wasn't interested. He wanted to know how to filter output through filters Oct 24 '15 at 22:50
  • He was also under the misapprehension that globbing is handled by commands themselves rather than by the shell.
    – roaima
    Oct 24 '15 at 22:52
  • @roaima Maybe. But things like grep work for other stuff as well, like cat /etc/passwd | egrep ":/home". I'm sure there are better examples. It's not the best question, though I can understand why such a question could be asked. Piping output through filters as he asked is something we do all the time. Oct 24 '15 at 23:05
  • egrep ':/home' /etc/passwd would be sufficient. But yes, pipelines are an important (if not essential) part of the Unix offering.
    – roaima
    Oct 24 '15 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.