If using while read loops for text processing in bash is bad...what should I do, then?

Question

I guess this may be a naive question but I can't get my head around so I felt like asking... I was searching for some solution to a problem, when I found this very interesting post about why is using [while|for] loops in bash considered bad practice. There is a very good explanation in the post (see the chosen answer) but I can't find anything that solves the issues that are discussed.

I searched extensively: I googled (or duckduckgo-ed) how to read a file in bash and all the results I am getting point towards a solution that, according to the above-mentioned post, is absolutely non-bash style and something that should be avoided. In particular, we have this:

while read line; do
  echo $line | cut -c3
done

and this:

for line in `cat file`; do
  foo=`echo $line | awk '{print $2}'`
  echo whatever $foo
done

that are indicated as very bad examples of shell scripting. At this point I am wondering, and this is the actual question: if the posted while loops should be avoided because they are bad practice and whatever...what am I supposed to do, instead?

EDIT: I see that I am already having comments/questions addressing the exact issue with the while loop, so I feel like to widen the question a bit. Basically, what I am understanding is that I need to dig deeper into bash commands, and that is the real thing that I should do. But, when one searches around, it looks like people are, in the general case, using and teaching bash in an improper way (as per my google-ing).

Answer 1

The point of the post you linked to is to explain that using bash to parse text files is a bad idea in general. It isn't specifically about using loops and there is nothing intrinsically wrong with shell loops in other contexts. Nobody is saying that a shell script with while is somehow bad. That other post is saying that you shouldn't try to parse text files using the shell and you should instead use other tools.

To clarify, when I say "using the shell" I mean using the shell's internal tools to open the file, extract the data and parse it. For example something like this:

while read number; do
  if [ $number -gt 10 ]; then
    echo "The number '$number' is greater than 10"
  else
    echo "The number '$number' is less than or equal to 10"
done < numbers.txt

Please read the answers at Why is using a shell loop to process text considered bad practice? for details on why this sort of thing is a bad idea. Here, I will only clarify that that post isn't arguing against shell loops in general, but against using shell loops (or the shell) for parsing files.

The reason you don't find suggestions for better ways of doing it with bash is that there are no good ways of doing this with bash or any other shell. No matter what you do, parsing text using a shell will be slow, cumbersome, and error prone.

Shells are primarily designed as a way of entering commands to be run by the computer. They can be used as scripting languages but, again, they are at their best when given commands to run and not when used instead of commands designed to handle text parsing.

Shells are tools and just like any other tool, they should be used for the purpose they were designed for. The problem is that many people have learned a little bit of shell scripting, so they have a tool, a "hammer". Because all they know is a hammer, every problem they encounter looks like a nail to them and they try and use their hammer on this nail. Sadly, parsing text is not something that the shell was designed to handle, it isn't a "nail", so using a "hammer" is just not a good idea.

So, the answer to "how should I read a file in bash" is very simply "you should not use bash and instead use a tool that is appropriate for the job".

Answer 2

The thing to avoid in your examples isn’t the looping, it’s the pointless use of multiple invocations of commands. It just happens that looping is one of the most common causes of useless invocations of commands in shell scripts (the other big one is not remembering to just use redirection).

Starting a new process is one of the most expensive operations possible on almost any system, so efficient scripts (and efficient code in general) keep the total number of processes to a minimum. This efficiency limitation is a large part of why inetd has fallen out of favor, and why many web servers default to starting a bunch of long-lived processes and handing connections to them as needed instead of spawning a process per connection on-demand.

Both of your examples can be reduced to starting a single process for the whole operation. The first one thus would become:

cut -c3

And the second would instead be:

awk '{print $2}'` < file

Not only are those more efficient, they’re also more readable.

This is not to say that looping in general is bad, just that a lot of things you might use it for in other languages don’t require it in shell scripts because the tools involved inherently process multiple lines or files. A good example of something it would be valid to use it for is handling of multiple attempts at doing something (assuming that the ‘something’ does not inherently support retries).

Answer 3

Instead of using a shell while or for loop that calls awk once for each line, just run awk once, giving it the filename as an argument. e.g.

awk '{print "whatever " $2}' file

same with cut:

cut -c3 file

If you need to do further processing in bash on each line returned by awk, your best option is to use command substitution to populate an array.

myarray=( $(awk '{print $2}' file) )

It's important to not double-quote the command substitution here because we want the shell's word-splitting to occur - each element of the array will be one "word" and, as awk's input is whitespace-separated and it only prints one field, it will output one "word" per line.

Alternatively, you can use the bash built-in readarray aka mapfile along with process substitution:

mapfile -t myarray < <(awk '{print $2}' file)

The mapfile/readarray variant is required if the input contains glob patterns like * in $2, otherwise the shell will try to expand the glob.

Once you have the data in an array, you can iterate over it with a for loop, e.g.:

for i in "${myarray[@]}"; do do_something_with "$i"; done

or pass it as args to another program or built-in:

printf "whatever %s\n" "${myarray[@]}"

Note, however, that you will almost always be better off doing any extra processing in awk. This may mean re-designing and re-writing your bash script so that most of the work is done in awk. Or rewriting the whole thing as an awk script if it turns out that bash isn't needed. ditto for perl. and python. and other languages.

shell is a good language for orchestrating other programs to process data and do actual work, but is terrible at doing data processing work itself - almost any other language would be better than shell for processing data.

If you find yourself moving data back and forth between shell and awk or some other language, that's a good sign that you need to rewrite the whole thing in awk (or whatever).

Answer 4

The other answers are correct, I just wonder if it's difficult to understand the concepts they are conveying. So I want to present you with a metaphor. Hopefully it makes sense:

You can consider the shell script as like a conductor. The conductor points at different musicians to indicate they should start or stop playing, play louder or softer, faster or slower. The conductor themselves doesn't step down into the orchestra to start playing an instrument; neither does the conductor "micro-manage" the musician by telling them how to turn the pages of their music sheets. The conductor trusts the musician to do that work themselves. The conductor just instructs the musicians when to play and how to play.

You shell script is like this. It is orchestrating a lot of other commands. It is not directly manipulating files at all. Even when you use a command like mv or cp, those commands are doing the actual work. They are like the musicians. The shell script says to the mv "musician" — move this here — and it does it. The shell script doesn't move the file itself. It lets the mv command do it.

Also, just like the conductor doesn't tell the musician when to turn the page, the shell script doesn't need to feed a file one line at a time to a command. It can give the whole file to the command and tell it what to do with it. Thus there is no need for a loop. The loop shouldn't be used to micro-manage the command, it should be used to orchestrate multiple commands over multiple files.

Don't forget that "commands" in shell scripts are not like commands in other languages — they are not keywords that form the shell script's language. Rather they are all separate programs that can be run from the shell. cp, for example is a program of its own. That program copies files. It has its own manual and a list of arguments you can pass to it. You can run cp from a shell (outside a script) and all you are doing is calling a program called cp. In the same way you could create your own programs (by creating shell scripts) and call them from within your shell script.

Hope that helped to explain it a bit.

Answer 5

I am writing this answer from a pragmatic perspective. I don't think using loops in Bash is necessarily bad.

A big benefit of using scripting languages like Bash is to realize tasks that come to your mind as quickly and easily as possible. Bash enables this by providing you the ability to combine small, efficient, general purpose programs like cat, cut, head, grep, tee, etc. via pipes. One would prefer using a Bash one-liner that makes use of a set of already efficient programs (like grep <something> input-file | cut -c3) rather than writing a program from scratch that does the same job, most of the case even it is slower.

If you roughly know what Bash can offer you, meaning you know sufficient amount of basic unix programs, Bash's piping features and programming language structures, and if it still is convenient to use loops, just do it.

Answer 6

There is absolutely nothing wrong with the examples you gave. No, they aren't using processing time efficiently. But that's fine in many circumstances. Only start optimizing once it's actually beneficial. And when you start to have a real need for optimization, you are probably best off dumping bash entirely and using a proper language instead. It's also worth it to learn how to mix bash with your high level language of choice so you can leverage the best of both worlds.