Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I consistently see answers quoting this link stating definitively "Don't parse ls!" This bothers me for a couple of reasons:

  1. It seems the information in that link has been accepted wholesale with little question, though I can pick out at least a few errors in casual reading.

  2. It also seems as if the problems stated in that link have sparked no desire to find a solution.

From the first paragraph:

...when you ask [ls] for a list of files, there's a huge problem: Unix allows almost any character in a filename, including whitespace, newlines, commas, pipe symbols, and pretty much anything else you'd ever try to use as a delimiter except NUL. ... ls separates filenames with newlines. This is fine until you have a file with a newline in its name. And since I don't know of any implementation of ls that allows you to terminate filenames with NUL characters instead of newlines, this leaves us unable to get a list of filenames safely with ls.

Bummer, right? How ever can we handle a newline terminated listed dataset for data that might contain newlines? Well, if the people answering questions on this websiite didn't do this kind of thing on a daily basis, I might think we were in some trouble.

The truth is though, most ls implementations actually provide a very simple api for parsing their output and we've all been doing it all along without even realizing it. Not only can you end a filename with null, you can begin one with null as well or with any other arbitrary string you might desire. What's more, you can assign these arbitrary strings per file-type. Please consider:

LS_COLORS='lc=\0:rc=:ec=\0\0\0:fi=:di=:' ls -l --color=always | cat -A
total 4$
drwxr-xr-x 1 mikeserv mikeserv 0 Jul 10 01:05 ^@^@^@^@dir^@^@^@/$
-rw-r--r-- 1 mikeserv mikeserv 4 Jul 10 02:18 ^@file1^@^@^@$
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 10 01:08 ^@file2^@^@^@$
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 10 02:27 ^@new$
line$
file^@^@^@$
^@

See this for more.

Now it's the next part of this article that really gets me though:

$ ls -l
total 8
-rw-r-----  1 lhunath  lhunath  19 Mar 27 10:47 a
-rw-r-----  1 lhunath  lhunath   0 Mar 27 10:47 a?newline
-rw-r-----  1 lhunath  lhunath   0 Mar 27 10:47 a space

The problem is that from the output of ls, neither you or the computer can tell what parts of it constitute a filename. Is it each word? No. Is it each line? No. There is no correct answer to this question other than: you can't tell.

Also notice how ls sometimes garbles your filename data (in our case, it turned the \n character in between the words "a" and "newline" into a ?question mark...

...

If you just want to iterate over all the files in the current directory, use a for loop and a glob:

for f in *; do
    [[ -e $f ]] || continue
    ...
done

The author calls it garbling filenames when ls returns a list of filenames containing shell globs and then recommends using a shell glob to retrieve a file list!

Consider the following:

printf 'touch ./"%b"\n' "file\nname" "f i l e n a m e" |
    . /dev/stdin
ls -1q

f i l e n a m e  
file?name

IFS="
" ; printf "'%s'\n" $(ls -1q)

'f i l e n a m e'
'file
name'

POSIX defines the -1 and -q ls operands so:

-q - Force each instance of non-printable filename characters and <tab>s to be written as the question-mark ( '?' ) character. Implementations may provide this option by default if the output is to a terminal device.

-1 - (The numeric digit one.) Force output to be one entry per line.

Globbing is not without its own problems - the ? matches any character so multiple matching ? results in a list will match the same file multiple times. That's easily handled.

Though how to do this thing is not the point - it doesn't take much to do after all and is demonstrated below - I was interested in why not. As I consider it, the best answer to that question has been accepted. I would suggest you try to focus more often on telling people what they can do than on what they can't. You're a lot less likely, as I think, to be proven wrong at least.

But why even try? Admittedly, my primary motivation was that others kept telling me I couldn't. I know very well that ls output is as regular and predictable as you could wish it so long as you know what to look for. Misinformation bothers me more than do most things.

The truth is, though, with the notable exception of both Patrick's and Wumpus Q. Wumbley's answers (despite the latter's awesome handle), I regard most of the information in the answers here as mostly correct - a shell glob is both more simple to use and generally more effective when it comes to searching the current directory than is parsing ls. They are not, however, at least in my regard, reason enough to justify either propagating the misinformation quoted in the article above nor are they acceptable justification to "never parse ls."

Please note that Patrick's answer's inconsistent results are mostly a result of him using zsh then bash. zsh - by default - does not word-split $(command substituted) results in a portable manner. So when he asks where did the rest of the files go? the answer to that question is your shell ate them. This is why you need to set the SH_WORD_SPLIT variable when using zsh and dealing with portable shell code. I regard his failure to note this in his answer as awfully misleading.

Wumpus's answer doesn't compute for me - in a list context the ? character is a shell glob. I don't know how else to say that.

In order to handle a multiple results case you need to restrict the glob's greediness. The following will just create a test base of awful file names and display it for you:

{ printf %b $(printf \\%04o `seq 0 127`) |
sed "/[^[-b]*/s///g
        s/\(.\)\(.\)/touch '?\v\2' '\1\t\2' '\1\n\2'\n/g" |
. /dev/stdin

echo '`ls` ?QUOTED `-m` COMMA,SEP'
ls -qm
echo ; echo 'NOW LITERAL - COMMA,SEP'
ls -m | cat
( set -- * ; printf "\nFILE COUNT: %s\n" $# )
}

OUTPUT

`ls` ?QUOTED `-m` COMMA,SEP
??\, ??^, ??`, ??b, [?\, [?\, ]?^, ]?^, _?`, _?`, a?b, a?b

NOW LITERAL - COMMA,SEP
?
 \, ?
     ^, ?
         `, ?
             b, [       \, [
\, ]    ^, ]
^, _    `, _
`, a    b, a
b

FILE COUNT: 12

Now I'll safe every character that isn't a /slash, -dash, :colon, or alpha-numeric character in a shell glob then sort -u the list for unique results. This is safe because ls has already safed-away any non printable characters for us. Watch:

for f in $(
        ls -1q |
        sed 's|[^-:/[:alnum:]]|[!-\\:[:alnum:]]|g' |
        sort -u | {
                echo 'PRE-GLOB:' >&2
                tee /dev/fd/2
                printf '\nPOST-GLOB:\n' >&2
        }
) ; do
        printf "FILE #$((i=i+1)): '%s'\n" "$f"
done

OUTPUT:

PRE-GLOB:
[!-\:[:alnum:]][!-\:[:alnum:]][!-\:[:alnum:]]
[!-\:[:alnum:]][!-\:[:alnum:]]b
a[!-\:[:alnum:]]b

POST-GLOB:
FILE #1: '?
           \'
FILE #2: '?
           ^'
FILE #3: '?
           `'
FILE #4: '[     \'
FILE #5: '[
\'
FILE #6: ']     ^'
FILE #7: ']
^'
FILE #8: '_     `'
FILE #9: '_
`'
FILE #10: '?
            b'
FILE #11: 'a    b'
FILE #12: 'a
b'

Below I approach the problem again but I use a different methodology. Remember that - besides \0null - the / ASCII character is the only byte forbidden in a pathname. I put globs aside here and instead combine the POSIX specified -d option for ls and the also POSIX specified -exec $cmd {} + construct for find. Because find will only ever naturally emit one / in sequence, the following easily procures a recursive and reliably delimited filelist including all dentry information for every entry. Just imagine what you might do with something like this:

#v#note: to do this fully portably substitute an actual newline \#v#
#v#for 'n' for the first sed invocation#v#
cd ..
find ././ -exec ls -1ldin {} + |
sed -e '\| *\./\./|{s||\n.///|;i///' -e \} |
sed 'N;s|\(\n\)///|///\1|;$s|$|///|;P;D'

###OUTPUT

152398 drwxr-xr-x 1 1000 1000        72 Jun 24 14:49
.///testls///

152399 -rw-r--r-- 1 1000 1000         0 Jun 24 14:49
.///testls/?
            \///

152402 -rw-r--r-- 1 1000 1000         0 Jun 24 14:49
.///testls/?
            ^///

152405 -rw-r--r-- 1 1000 1000         0 Jun 24 14:49
.///testls/?
        `///
...

ls -i can be very useful - especially when result uniqueness is in question.

ls -1iq | 
sed '/ .*/s///;s/^/-inum /;$!s/$/ -o /' | 
tr -d '\n' | 
xargs find

These are just the most portable means I can think of. With GNU ls you could do:

ls --quoting-style=WORD

And last, here's a much simpler method of parsing ls that I happen to use quite often when in need of inode numbers:

ls -1iq | grep -o '^ *[0-9]*'

That just returns inode numbers - which is another handy POSIX specified option.

share|improve this question
7  
@mikeserv Ok I did. Shell glob is 2.48 times faster. time bash -c 'for i in {1..1000}; do ls -R &>/dev/null; done' = 3.18s vs time bash -c 'for i in {1..1000}; do echo **/* >/dev/null; done' = 1.28s –  Patrick May 12 at 4:05
20  
In regards to your most recent update, please stop relying on visual output as determining that your code works. Pass your output to an actual program and have the program try and perform an operation on the file. This is why I was using stat in my answer, as it actually checks that each file exists. Your bit at the bottom with the sed thing does not work. –  Patrick May 12 at 4:20
26  
You can't be serious. How can jumping through all the hoops your question describes be easier or simpler or in any way better than simply not parsing ls in the first place? What you're describing is very hard. I'll need to deconstruct it to understand all of it and I'm a relatively competent user. You can't possibly expect your average Joe to be able to deal with something like this. –  terdon May 12 at 4:40
27  
-1 for using a question to pick an argument. All of the reasons parsing ls output is wrong were covered well in the original link (and in plenty of other places). This question would have been reasonable if OP were asking for help understanding it, but instead OP is simply trying to prove his incorrect usage is ok. –  R.. May 12 at 13:05
8  
@mikeserv It's not just that parsing ls is bad. Doing for something in $(command) and relying on word-splitting to get accurate results is bad for the large majority of command's which don't have simple output. –  BroSlow May 13 at 14:53

8 Answers 8

up vote 82 down vote accepted
+150

I am not at all convinced of this, but let's suppose for the sake of argument that you could, if you're prepared to put in enough effort, parse the output of ls reliably, even in the face of an "adversary" — someone who knows the code you wrote and is deliberately choosing filenames designed to break it.

Even if you could do that, it would still be a bad idea.

Bourne shell is not a good language. It should not be used for anything complicated, unless extreme portability is more important than any other factor (e.g. autoconf).

I claim that if you're faced with a problem where parsing the output of ls seems like the path of least resistance for a shell script, that's a strong indication that whatever you are doing is too complicated for shell and you should rewrite the entire thing in Perl or Python. Here's your last program in Python:

import os, sys
for subdir, files, dirs in os.walk("."):
    for f in dirs + files:
      ino = os.lstat(os.path.join(subdir, f)).st_ino
      sys.stdout.write("%d %s %s\n" % (ino, subdir, f))

This has no issues whatsoever with unusual characters in filenames -- the output is ambiguous in the same way the output of ls is ambiguous, but that wouldn't matter in a "real" program (as opposed to a demo like this), which would use the result of os.path.join(subdir, f) directly.

Equally important, and in stark contrast to the thing you wrote, it will still make sense six months from now, and it will be easy to modify when you need it to do something slightly different. By way of illustration, suppose you discover a need to exclude dotfiles and editor backups, and to process everything in alphabetical order by basename:

import os, sys
filelist = []
for subdir, files, dirs in os.walk("."):
    for f in dirs + files:
        if f[0] == '.' || f[-1] == '~': continue
        ino = os.lstat(os.path.join(subdir, f)).st_ino
        filelist.append((f, subdir, ino))

filelist.sort(key = lambda x: x[0])
for f, path, ino in filelist: 
   sys.stdout.write("%d %s %s\n" % (ino, subdir, f))
share|improve this answer
2  
No recursion, just nested for-loops. os.walk is doing some seriously heavy lifting behind the scenes, but you don't have to worry about it any more than you have to worry about how ls or find work internally. –  Zack May 12 at 23:04
3  
Technically, os.walk returns a generator object. Generators are Python's version of lazy lists. Every time the outer for-loop iterates, the generator is invoked and "yields" the contents of another subdirectory. Equivalent functionality in Perl is File::Find, if that helps. –  Zack May 12 at 23:12
1  
You should be aware that I 100% agree with the document you are criticizing and with Patrick and Terdon's answers. My answer was intended to provide an additional, independent reason to avoid parsing ls output. –  Zack May 13 at 17:02
2  
I should note that in various comments, mikeserv's primary reason for parsing ls was that he can do some additional preprocessing (like sorting or filtering with grep) before the traversal. This alternative does not currently do that. –  Izkata May 13 at 18:36
7  
@MilesRout Shell is not a good any kind of language. Even in situations that it should be good at (e.g. /etc/init.d scripts, which are what it was designed for, insomuch as it was designed at all) the obvious way to do ... everything ... will leave you with subtle bugs, the correct way to do everything is tedious, and you have to read character by fucking character, looking for things that aren't there, to notice the difference. And I don't even want to talk about how tiny the true portable subset is. –  Zack May 13 at 23:33

That link is referenced a lot because the information is completely accurate, and it has been there for a very long time.

 


ls replaces non-printable characters with glob characters yes, but those characters aren't in the actual filename. Why does this matter? 2 reasons:

  1. If you pass that filename to a program, that filename doesn't actually exist. It would have to expand the glob to get the real file name.
  2. The file glob might match more than one file.

For example:

$ touch a$'\t'b
$ touch a$'\n'b
$ ls -1
a?b
a?b

Notice how we have 2 files which look exactly the same. How are you going to distinguish them if they both are represented as a?b?


The author calls it garbling filenames when ls returns a list of filenames containing shell globs and then recommends using a shell glob to retrieve a file list!

There is a difference here. When you get a glob back, as shown, that glob might match more than one file. However when you iterate through the results matching a glob, you get back the exact file, not a glob.

For example:

$ for file in *; do printf '%s' "$file" | xxd; done
0000000: 6109 62                                  a.b
0000000: 610a 62                                  a.b

Notice how the xxd output shows that $file contained the raw charaters \t and \n, not ?.

If you use ls, you get this instead:

for file in $(ls -1q); do printf '%s' "$file" | xxd; done
0000000: 613f 62                                  a?b
0000000: 613f 62                                  a?b

"I'm going to iterate anyway, why not use ls?"

Your example you gave doesn't actually work. It looks like it works, but it doesn't.

I'm referring to this:

 for f in $(ls -1q | tr " " "?") ; do [ -f "$f" ] && echo "./$f" ; done

I've created a directory with a bunch of file names:

$ for file in *; do printf '%s' "$file" | xxd; done
0000000: 6120 62                                  a b
0000000: 6120 2062                                a  b
0000000: 61e2 8082 62                             a...b
0000000: 61e2 8083 62                             a...b
0000000: 6109 62                                  a.b
0000000: 610a 62                                  a.b

When I run your code, I get this:

$ for f in $(ls -1q | tr " " "?") ; do [ -f "$f" ] && echo "./$f" ; done
./a b
./a b

Where'd the rest of the files go?

Let's try this instead:

$ for f in $(ls -1q | tr " " "?") ; do stat --format='%n' "./$f"; done 
stat: cannot stat ‘./a?b’: No such file or directory
stat: cannot stat ‘./a??b’: No such file or directory
./a b
./a b
stat: cannot stat ‘./a?b’: No such file or directory
stat: cannot stat ‘./a?b’: No such file or directory

Now lets use an actual glob:

$ for f in *; do stat --format='%n' "./$f"; done                        
./a b
./a  b
./a b
./a b
./a b
./a
b

With bash

The above example was with my normal shell, zsh. When I repeat the procedure with bash, I get another completely different set of results with your example:

Same set of files:

$ for file in *; do printf '%s' "$file" | xxd; done
0000000: 6120 62                                  a b
0000000: 6120 2062                                a  b
0000000: 61e2 8082 62                             a...b
0000000: 61e2 8083 62                             a...b
0000000: 6109 62                                  a.b
0000000: 610a 62                                  a.b

Radically different results with your code:

for f in $(ls -1q | tr " " "?") ; do stat --format='%n' "./$f"; done 
./a b
./a b
./a b
./a b
./a
b
./a  b
./a b
./a b
./a b
./a b
./a b
./a b
./a
b
./a b
./a b
./a b
./a b
./a
b

With a shell glob, it works perfectly fine:

$ for f in *; do stat --format='%n' "./$f"; done  
./a b
./a  b
./a b
./a b
./a b
./a
b

The reason bash behaves this way goes back to one of the points I made at the beginning of the answer: "The file glob might match more than one file".

ls is returning the same glob (a?b) for several files, so each time we expand this glob, we get every single file that matches it.


How to recreate the list of files I was using:

touch 'a b' 'a  b' a$'\xe2\x80\x82'b a$'\xe2\x80\x83'b a$'\t'b a$'\n'b

The hex code ones are UTF-8 NBSP characters.

share|improve this answer
4  
@mikeserv actually his solution doesn't return a glob. I just updated my answer to clarify that point. –  Patrick May 12 at 2:02
11  
"Not the rest"? It's inconsistent behavior, and unexpected results, how is that not a reason? –  Patrick May 12 at 2:32
2  
@mikeserv You could avoid the duplicates with something like for f in $(ls -1q | tr " " "?" | sed 's/^/"/; s/$/"/') ; do echo "$f"; done. But why not just for f in *; do echo "$f"; done? –  terdon May 12 at 2:32
6  
@mikeserv Did you not see my comment on your question? Shell globbing is 2.5 times faster than ls. I also requested that you test your code as it does not work. What does zsh have to do with any of this? –  Patrick May 12 at 4:29
16  
@mikeserv No, it all still applies even to bash. Though I'm done with this question as you're not listening to what I'm saying. –  Patrick May 12 at 5:37

Let's try and simplify a little:

$ touch a$'\n'b a$'\t'b 'a b'
$ ls
a b  a?b  a?b
$ IFS="
"
$ set -- $(ls -1q | uniq)
$ echo "Total files in shell array: $#"
Total files in shell array: 4

See? That's already wrong right there. There are 3 files but bash is reporting 4. This is because the set is being given the globs generated by ls which are expanded by the shell before being passed to set. Which is why you get:

$ for x ; do
>     printf 'File #%d: %s\n' $((i=$i+1)) "$x"
> done
File #1: a b
File #2: a b
File #3: a    b
File #4: a
b

Or, if you prefer:

$ printf ./%s\\0 "$@" |
> od -A n -c -w1 |
> sed -n '/ \{1,3\}/s///;H
> /\\0/{g;s///;s/\n//gp;s/.*//;h}'
./a b
./a b
./a\tb
./a\nb

The above was run on bash 4.2.45.

share|improve this answer
    
I upvoted this. It's good to see your own code bite you. But just because I got it wrong doesn't mean it can't be done right. I showed you a very simple way to do it this morning with ls -1qRi | grep -o '^ *[0-9]*' - that's parsing ls output, man, and it's the fastest and best way of which I know to get a list of inode numbers. –  mikeserv May 12 at 22:56
16  
@mikeserv: It could be done right, if you have the time and patience. But the fact is, it is inherently error-prone. You yourself got it wrong. while arguing about its merits! That's a huge strike against it, if even the one person fighting for it fails to do it correctly. And chances are, you'll probably spend still more time getting it wrong before you get it right. I dunno about you, but most people have better to do with their time than fiddle around for ages with the same line of code. –  cHao May 13 at 1:06
    
@cHao - i didnt argue its merits - i protested its propaganda. –  mikeserv May 13 at 1:47
7  
@mikeserv: The arguments against it are well-founded and well-deserved. Even you have shown them to be true. –  cHao May 13 at 1:50
    
@cHao - i disagree. There is a not-so-fine line between a mantra and a wisdom. –  mikeserv May 13 at 1:51

The output of ls -q isn't a glob at all. It uses ? to mean "There is a character here that can't be displayed directly". Globs use ? to mean "Any character is allowed here".

Globs have other special characters (* and [] at least, and inside the [] pair there are more). None of those are escaped by ls -q.

$ touch x '[x]'
$ ls -1q
[x]
x

If you treat the ls -1q output there are a set of globs and expand them, not only will you get x twice, you'll miss [x] completely. As a glob, it doesn't match itself as a string.

ls -q is meant to save your eyes and/or terminal from crazy characters, not to produce something that you can feed back to the shell.

share|improve this answer

The answer is simple: the special cases of ls you have to handle outweigh any possible use. These special cases can be averted if you don't parse ls output.

The mantra here is never trust the user filesystem (the equivalent to never trust user input). If there's a method that will work always, being 100% certain, it should be the method you prefer even if ls do the same but with less certainly. I won't go into technical details since those were covered by terdon and Patrick extensively, but I know that the risks of using ls in an important (and maybe expensive) transaction where my neck (job or prestige) is out there, I will prefer any solution that doesn't have a grade of uncertainly if it can be adverted.

I know people will prefer some risk over certainly, but I've filled a bug report.

share|improve this answer

The reason people say never do something isn't necessarily because it absolutely positively cannot be done correctly. We may be able to do so, but it may be more complicated, less efficient both space- or time-wise. For example it would be perfectly fine to say "Never build a large e-commerce backend in x86 assembly".

So now to the issue at hand: As you've demonstrated you can create a solution that parses ls and gives the right result - so correctness isn't an issue.

Is it more complicated? Yes, but we can hide that behind a helper function.

So now to efficiency:

Space-efficiency: Your solution relies on uniq to filter out duplicates, consequently we cannot generate the results lazily. So either O(1) vs. O(n) or both have O(n).

Time-efficiency: Best case uniq uses a hashmap approach so we still have a O(n) algorithm in the number of elements procured, probably though it's O(n log n).

Now the real problem: While your algorithm is still not looking too bad I was really careful to use elements procured and not elements for n. Because that does make a big difference. Say you have a file \n\n that will result in a glob for ?? so match every 2 character file in the listing. Funnily if you have another file \n\r that will also result in ?? and also return all 2 character files.. see where this is going? Exponential instead of linear behavior certainly qualifies as "worse runtime behavior".. it's the difference between a practical algorithm and one you write papers in theoretical CS journals about.

Everybody loves examples right? Here we go. Make a folder called "test" and use this python script in the same directory where the folder is.

#!/usr/bin/env python3
import itertools
dir = "test/"
filename_length = 3
options = "\a\b\t\n\v\f\r"

for filename in itertools.product(options, repeat=filename_length):
        open(dir + ''.join(filename), "a").close()

Only thing this does is generate all products of length 3 for 7 characters. High school math tells us that ought to be 343 files. Well that ought to be really quick to print, so let's see:

time for f in *; do stat --format='%n' "./$f" >/dev/null; done
real    0m0.508s
user    0m0.051s
sys 0m0.480s

Now let's try your first solution, because I really can't get this

eval set -- $(ls -1qrR ././ | tr ' ' '?' |
sed -e '\|^\(\.\{,1\}\)/\.\(/.*\):|{' -e \
        's//\1\2/;\|/$|!s|.*|&/|;h;s/.*//;b}' -e \
        '/..*/!d;G;s/\(.*\)\n\(.*\)/\2\1/' -e \
        "s/'/'\\\''/g;s/.*/'&'/;s/?/'[\"?\$IFS\"]'/g" |
uniq)

thing here to work on Linux mint 16 (which I think speaks volumes for the usability of this method).

Anyhow since the above pretty much only filters the result after it gets it, the earlier solution should be at least as quick as the later (no inode tricks in that one- but those are unreliable so you'd give up correctness).

So now how long does

time for f in $(ls -1q | tr " " "?") ; do stat --format='%n' "./$f" >/dev/null; done

take? Well I really don't know, it takes a while to check 343^343 file names - I'll tell you after the heat death of the universe.

share|improve this answer

preface and original answer's rationaleupdated on 2015-05-18

mikeserv (the OP) stated in latest update to his question: "I do consider it a shame though that I first asked this question to point out a source of misinformation, and, unfortunately, the most upvoted answer here is in large part misleading."

Well, okay; I feel it was rather a shame that I spent so much time trying to figure out how to explain my meaning only to find that as I re-read the question. This question ended up "[generating] discussion rather than answers" and ended up weighing in at ~18K of text (for the question alone, just to be clear) which would be long even for a blog post.

But StackExchange is not your soapbox, and it's not your blog. But you, in effect, have used it as at least bit of both. People ended up spending a lot of time answering your "To-Point-Out" instead of answering people's actual questions. At this point I will be flagging the question as not a good fit for our format, given that the OP has stated explicitly that it wasn't even intended to be a question at all.

At this point I'm not sure whether my answer was to the point, or not; probably not, but it was directed at some of your questions, and maybe it can be a useful answer to someone else; beginners take heart, some of those "do not"s turn into "do sometimes" once you get more experienced. :)

[original, with clarifications]
have gone far over the time I should spend on this; I apologize for any remaining rough edges

Just to be clear on what question I'm answering; I was originally going to gather quotes from the question but I think a summary suffices: I may have very well just over-emphasized these, but I think they are relevant. Stated in several different ways, the original post asked questions like why articles would say "Don't parse ls output" or "You should never parse ls output" or similar. [note: not direct quotes].

So, basically my clarification is this: these are all basically a single idiom, but phrased in slightly different ways, stating an imperative together with 'always' or 'never' which, nonetheless are intended as general rules to be used; and when you're a beginner with some subject matter, generally it's best to always follow the general rules without some guidance from someone more experienced; and with rising skill and experience you will be able to, yourself, determine when and if these things are appropriate, and generally you should know when that is because you'll understand all that goes into the problem.

And that's when you choose to do things, perhaps in violation of the "The Rules". But that doesn't make them not general rules. And, it doesn't mean that it would be better to tell a beginner that "sometimes" it's okay to parse ls output, because, it's not. Put your pawns in the center and in the opening one piece, one move, and castle at the earliest opportunity. (Whoops, getting tired, wrong subject.)

General rule general format examples: "You should not ever do X", "Never do X!", "Don't do X!", or "You should always do Y!", and so forth.

In these cases, the general statements are idiomatic uses of phrases that seem to be stating rules valid in all times and places and for all people; but, instead, they're a general rule (or, "rule of thumb", "the basics", etc.) stated in a way that is likely appropriate for the people that read those kind of articles, but those rules are not saying anything about whether an expert might choose to do it when necessary/appropriate. Goto is an example; there's been a big debate on whether they are harmful. (Yeah, don't use gotos. ;D)

To repeat: the feature of a general rule is that they are stated as modal propositions, yet aren't equivalent to the same statements in modal logic (thus, "idiomatic", rather than "literal") and experts break general rules all the time under the right conditions. Instead, it's a way of saying that, unless you have the ability to tell that this statement is incorrect in a particular case, and prove to yourself that you are right, then you should follow the rule.

And this rule is a good one. It's not just that you have to be very good with shell scripting to know whether it can be broken, in some particular case. It's, also, that it's takes just as much skill to tell you got it wrong when you are trying to break it in testing! And, I say confidently that a very large majority likely audience of these articles (that give advice like "Don't parse the output of ls!") can't do those things, and those that do have such skill likely can figure it out on their own.

Just look at this question, and how even people that probably do have the skill thought it was a bad call to do so; and how much effort the author of the question spent just getting to a point of the current best example! I guarantee you on a problem that hard, 99% of the people out there would get it wrong, and with potentially very bad results! Even if the method you've decided on turns out to work; until it (or another) ls parsing idea becomes adopted by IT/developer folk, withstands a lot of testing, especially the test of time, and finally graduates to 'common technique' status, a lot of people might try and get it wrong with disastrous consequences.

So, I do reiterate: especially in this case, that is why "never" is decidedly the right way to phrase it.

[UPDATE 2014-05-18: clarified reasoning for answer (above) to respond to a comment from OP; the following addition is in response to the OP's additions to the question from yesterday]

† I originally intended this solely as a clarification on my original; but decided on other additions upon reflection

‡ see http://unix.stackexchange.com/tour for guidelines on posts

share|improve this answer
    
Never isn't idiomatic. This is not an answer to anything. –  mikeserv May 17 at 17:52
    
Hmm. Well, I didn't know whether this answer would be satisfying but I absolutely didn't expect it to be controversial. And, I didn't (mean to) argue that 'never' was per se idiomatic; but that "Never do X!" is an idiomatic use. I see two general cases that can show that 'Never/don't parse ls!' is correct advice: 1. demonstrate (to your satisfaction) that every use-case where one might parse ls output has another available solution, superior in some way, without doing so. 2. show that, in the cited cases, the statement is not a literal one. –  shelleybutterfly May 18 at 6:50
    
Looking at your question again, I see that you first mention "don't ..." rather than "never ..." which is well into your analysis, so I'll clarify on that point as well. At this point there's already a solution of the first type, which is apparently demonstrated/explained to your satisfaction, so I won't delve into there much. But I'll try and clarify my answer a bit: like I say, I wasn't trying to be controversial (or confrontational!) but to point out how those statements are generally intended. –  shelleybutterfly May 18 at 6:53
    
I actually posted it since I didn't see it given as an explanation; and, since I have more than once had someone hung up on my choice of a solution because 'but "everyone" says never to do that!' when I made a (non-conventional) judgement that it was nonetheless appropriate given the circumstances. [example: "You can't/shouldn't use C++ in real-time safety-critical software!"] The more thoroughly you understand something, and the more experience you have with it, the more often you will see a superior solution that 'breaks the rules', but that doesn't mean a beginner should do so willy-nilly. –  shelleybutterfly May 18 at 7:10
1  
Well, i reversed my downvote because, at the very least, youre right about the flagging thing. Ill try to clean it up tonight or tomorrow. My thought is i'll move most of the code examples to an answer i guess. But it still doesnt, as far as im concerned, excuse the inaccuracies in that oft-cited blog post. I wish people would stop citing the bash manual altogether - at least not til after theyve cited the POSIX specs... –  mikeserv May 18 at 17:44

Is it possible to parse the output of ls in certain cases? Sure. The idea of extracting a list of inode numbers from a directory is a good example - if you know that your implementation's ls supports -q, and therefore each file will produce exactly one line of output, and all you need are the inode numbers, parsing them out of ls -Rai1q output is certainly a possible solution. Of course, if the author hadn't seen advice like "Never parse the output of ls" before, he probably wouldn't think about filenames with newlines in them, and would probably leave off the 'q' as a result, and the code would be subtly broken in that edge case - so, even in cases where parsing ls's output is reasonable, this advice is still useful.

The broader point is that, when a newbie to shell scripting tries to have a script figure out (for instance) what's the biggest file in a directory, or what's the most recently modified file in a directory, his first instinct is to parse ls's output - understandable, because ls is one of the first commands a newbie learns.

Unfortunately, that instinct is wrong, and that approach is broken. Even more unfortunately, it's subtly broken - it will work most of the time, but fail in edge cases that could perhaps be exploited by someone with knowledge of the code.

The newbie might think of ls -s | sort -n | tail -n 1 | awk '{print $2}' as a way to get the biggest file in a directory. And it works, until you have a file with a space in the name.

OK, so how about ls -s | sort -n | tail -n 1 | sed 's/[^ ]* *[0-9]* *//'? Works fine until you have a file with a newline in the name.

Does adding -q to ls's arguments help when there's a newline in the filename? It might look like it does, until you have 2 different files that contain a non-printable character in the same spot in the filename, and then ls's output doesn't let you distinguish which of those was biggest. Worse, in order to expand the "?", he probably resorts to his shell's eval - which will cause problems if he hits a file named, for instance,

foo`/tmp/malicious_script`bar

Does --quoting-style=shell help (if your ls even supports it)? Nope, still displays ? for nonprintable characters, so it's still ambiguous which of multiple matches was the biggest. --quoting-style=literal? Nope, same. --quoting-style=locale or --quoting-style=c might help if you just need to print the name of the biggest file unambiguously, but probably not if you need to do something with the file afterwards - it would be a bunch of code to undo the quoting and get back to the real filename so that you can pass it to, say, gzip.

And at the end of all that work, even if what he has is safe and correct for all possible filenames, it's unreadable and unmaintainable, and could have been done much more easily, safely, and readably in python or perl or ruby.

Or even using other shell tools - off the top of my head, I think this ought to do the trick:

find . -type f -printf "%s %f\0" | sort -nz | awk 'BEGIN{RS="\0"} END{sub(/[0-9]* /, "", $0); print}'

And ought to be at least as portable as --quoting-style is.

share|improve this answer
    
Oh true about the size - i probably could do that if i tried - should i? Im kinda tired or this whole thing - i like your answer because you dont say can't or dont or never but actually give examples of maybe why not and comparable how else - thank you. –  mikeserv May 16 at 16:44
    
I think if you tried, you'd discover it's much harder than you think. So, yes, I'd recommend trying. I'll be happy to keep giving filenames that will break for you as long as I can think of them. :) –  godlygeek May 16 at 16:50
    
Comments are not for extended discussion; this conversation has been moved to chat. –  terdon Aug 23 at 10:24
    
@mikeserv and godlygeek, I have moved this comment thread to chat. Please don't have long discussions like this in the comments, that's what chat is for. –  terdon Aug 23 at 10:28

protected by slm May 21 at 3:05

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.