14

I guess this may be a naive question but I can't get my head around so I felt like asking... I was searching for some solution to a problem, when I found this very interesting post about why is using [while|for] loops in bash considered bad practice. There is a very good explanation in the post (see the chosen answer) but I can't find anything that solves the issues that are discussed.

I searched extensively: I googled (or duckduckgo-ed) how to read a file in bash and all the results I am getting point towards a solution that, according to the above-mentioned post, is absolutely non-bash style and something that should be avoided. In particular, we have this:

while read line; do
  echo $line | cut -c3
done

and this:

for line in `cat file`; do
  foo=`echo $line | awk '{print $2}'`
  echo whatever $foo
done

that are indicated as very bad examples of shell scripting. At this point I am wondering, and this is the actual question: if the posted while loops should be avoided because they are bad practice and whatever...what am I supposed to do, instead?

EDIT: I see that I am already having comments/questions addressing the exact issue with the while loop, so I feel like to widen the question a bit. Basically, what I am understanding is that I need to dig deeper into bash commands, and that is the real thing that I should do. But, when one searches around, it looks like people are, in the general case, using and teaching bash in an improper way (as per my google-ing).

12
  • 12
    Regarding your examples: use cut or awk on the file directly. In general: There is no general answer.
    – pLumo
    Aug 9, 2021 at 14:31
  • 2
    The other question shows the alternatives when it's explaining why you shouldn't use shell loops.
    – Barmar
    Aug 10, 2021 at 14:32
  • 3
    I suggest to change the title. As it stands now, people see "If using while loops in bash is bad ..." in places like the Hot Network Questions. Things like this lead to public myths that ... using while loops in bash is bad, and might have bad influence on bash newbies. Which is nonsense. The question you linked shows a particular usage of while loops which is indeed bad, but the problem is not the while loop itself, at all, but directly translating programming practices from lower level general languages and ignoring the piping mechanism of shells.
    – AnoE
    Aug 11, 2021 at 6:27
  • 2
    Perfect, thanks. ;)
    – AnoE
    Aug 11, 2021 at 6:54
  • 2
    I rolled back your change, gabt. @AnoE please dont' tell users to make a change that completely alters the question being asked after people have already posted answers. The change you suggested renders existing answers irrelevant and also makes this question a direct duplicate of the one already linked. The answers there perfectly explain why it's a bad idea to use the shell for this and also cover the safest way of doing it.
    – terdon
    Aug 11, 2021 at 7:59

6 Answers 6

37

The point of the post you linked to is to explain that using bash to parse text files is a bad idea in general. It isn't specifically about using loops and there is nothing intrinsically wrong with shell loops in other contexts. Nobody is saying that a shell script with while is somehow bad. That other post is saying that you shouldn't try to parse text files using the shell and you should instead use other tools.

To clarify, when I say "using the shell" I mean using the shell's internal tools to open the file, extract the data and parse it. For example something like this:

while read number; do
  if [ $number -gt 10 ]; then
    echo "The number '$number' is greater than 10"
  else
    echo "The number '$number' is less than or equal to 10"
done < numbers.txt

Please read the answers at Why is using a shell loop to process text considered bad practice? for details on why this sort of thing is a bad idea. Here, I will only clarify that that post isn't arguing against shell loops in general, but against using shell loops (or the shell) for parsing files.

The reason you don't find suggestions for better ways of doing it with bash is that there are no good ways of doing this with bash or any other shell. No matter what you do, parsing text using a shell will be slow, cumbersome, and error prone.

Shells are primarily designed as a way of entering commands to be run by the computer. They can be used as scripting languages but, again, they are at their best when given commands to run and not when used instead of commands designed to handle text parsing.

Shells are tools and just like any other tool, they should be used for the purpose they were designed for. The problem is that many people have learned a little bit of shell scripting, so they have a tool, a "hammer". Because all they know is a hammer, every problem they encounter looks like a nail to them and they try and use their hammer on this nail. Sadly, parsing text is not something that the shell was designed to handle, it isn't a "nail", so using a "hammer" is just not a good idea.

So, the answer to "how should I read a file in bash" is very simply "you should not use bash and instead use a tool that is appropriate for the job".

8
  • 6
    I find that answer confusing and a bit misleading. cut -c3 < file and while read line; do echo $line | cut -c 3; done are two samples of bash code (bash being a shell, a tool whose purpose is to run other tools), the first one running one invocation of one text utility, the second one 3 incorrect invocations of 2 clunky tools and a text processing utility for each line of the file in a loop. Shells are designed to run tools some of which are good at processing text, what it being discussed here is how and which tools and how, not whether bash should be used to invoke them. Aug 9, 2021 at 17:58
  • 4
    @StéphaneChazelas sorry, which answer? This one? The OP had misunderstood your answer here as saying that using shell loops is bad in general. I am clarifying that no, the point of your answer was that using shells to parse text files is a bad idea, loops are not the issue. Using shells to invoke other tools is fine, of course.
    – terdon
    Aug 9, 2021 at 18:02
  • 3
    Sorry, I meant "this" answer (yours). The problem is not about using shells to process text, but using shells that way (with a loop calling many invocations of clunky tools, and here incorrectly) as opposed to one invocation of a text processing tool (or of several tools cooperating in a pipeline). Aug 9, 2021 at 18:07
  • 3
    @StéphaneChazelas yes. Also with using shells at all though. I mean things like if [[ $line =~ $regex ]] and the like, and pitfalls with while read etc. Far better to use the shell to call other tools that can handle this sort of thing better. I am addressing someone who wants to write a parser using the shell as a scripting language instead of using something like awk or sed or perl or python etc. Even in the best case scenario, as you explain so well in your answer, the shell approach will be horribly slow compared to anything else.
    – terdon
    Aug 9, 2021 at 18:12
  • 3
    @terdon: StéphaneChazelas is objecting to the way you use phrases such as "using bash to parse text files": you're using them to mean something like while read line; do echo $line | cut -c 3; done as opposed to something like cut -c3 < file, but he contends that such phrases do not distinguish between those examples, because both are examples of using Bash and cut. He's not disagreeing with your point that one of these is a bad idea and one of them is a good idea, he's just objecting to the phrasing you're using to convey that point.
    – ruakh
    Aug 10, 2021 at 8:08
13

The thing to avoid in your examples isn’t the looping, it’s the pointless use of multiple invocations of commands. It just happens that looping is one of the most common causes of useless invocations of commands in shell scripts (the other big one is not remembering to just use redirection).

Starting a new process is one of the most expensive operations possible on almost any system, so efficient scripts (and efficient code in general) keep the total number of processes to a minimum. This efficiency limitation is a large part of why inetd has fallen out of favor, and why many web servers default to starting a bunch of long-lived processes and handing connections to them as needed instead of spawning a process per connection on-demand.

Both of your examples can be reduced to starting a single process for the whole operation. The first one thus would become:

cut -c3

And the second would instead be:

awk '{print $2}'` < file

Not only are those more efficient, they’re also more readable.

This is not to say that looping in general is bad, just that a lot of things you might use it for in other languages don’t require it in shell scripts because the tools involved inherently process multiple lines or files. A good example of something it would be valid to use it for is handling of multiple attempts at doing something (assuming that the ‘something’ does not inherently support retries).

6
  • 2
    so, the idea here is that the while loop in the original question calls many commands, i.e. grep, or echo..., for each loop. While (!), if I use awk then pipe it's output to grep, sed, or whatever, this would be much more efficient (and readable and everything).
    – gabt
    Aug 10, 2021 at 6:44
  • 2
    @gabt Essentially yes, both in terms of the number of processes involved, and possibly in terms of memory usage (though that’s less clear and really dependent on a bunch of other factors). In general, almost all traditional UNIX command-line tools are designed to process whole files at a time, operating on a line-by-line basis, so it’s relatively rare that you need a loop in shell script to actually do this type of thing. Aug 10, 2021 at 11:38
  • 2
    @gabt You can also considering writing a perl script that reads a file line-by-line in a loop, does the same transformations that sed, awk or whatever would do, but using perl internal functions and not by calling external programs and writes the modified line back to output file. Perl is a language designed just for processing text files. And if you need, you can also call system commands like cp from within Perl.
    – raj
    Aug 10, 2021 at 13:43
  • 1
    @raj, yes I am actually reading around to understand which tool is worth learning to work with files and I surely considered perl. There are way too many tools and it's not that easy to choose one that works for everything (also I don't think such a thing exists). But, at least, I am now moving from bad shell scripting to, hopefully, something better.
    – gabt
    Aug 10, 2021 at 14:14
  • 3
    @gabt My personal suggestion would be to look to Python. It’s designed for more generalized usage than PERL was, is far more powerful than shell script, and is much easier to learn than either. Aug 10, 2021 at 17:25
13

Instead of using a shell while or for loop that calls awk once for each line, just run awk once, giving it the filename as an argument. e.g.

awk '{print "whatever " $2}' file

same with cut:

cut -c3 file

If you need to do further processing in bash on each line returned by awk, your best option is to use command substitution to populate an array.

myarray=( $(awk '{print $2}' file) )

It's important to not double-quote the command substitution here because we want the shell's word-splitting to occur - each element of the array will be one "word" and, as awk's input is whitespace-separated and it only prints one field, it will output one "word" per line.

Alternatively, you can use the bash built-in readarray aka mapfile along with process substitution:

mapfile -t myarray < <(awk '{print $2}' file)

The mapfile/readarray variant is required if the input contains glob patterns like * in $2, otherwise the shell will try to expand the glob.

Once you have the data in an array, you can iterate over it with a for loop, e.g.:

for i in "${myarray[@]}"; do do_something_with "$i"; done

or pass it as args to another program or built-in:

printf "whatever %s\n" "${myarray[@]}"

Note, however, that you will almost always be better off doing any extra processing in awk. This may mean re-designing and re-writing your bash script so that most of the work is done in awk. Or rewriting the whole thing as an awk script if it turns out that bash isn't needed. ditto for perl. and python. and other languages.

shell is a good language for orchestrating other programs to process data and do actual work, but is terrible at doing data processing work itself - almost any other language would be better than shell for processing data.

If you find yourself moving data back and forth between shell and awk or some other language, that's a good sign that you need to rewrite the whole thing in awk (or whatever).

12
  • 9
    @gabt the shell is a great tool for manipulating files, yes. Not their contents, usually, but the files themselves (think copying, moving, organizing into directories etc.). But it is a horrible tool for parsing the files or doing anything with strings or fields.
    – terdon
    Aug 9, 2021 at 15:03
  • 2
    @gabt - yep, what terdon said.
    – cas
    Aug 9, 2021 at 15:05
  • 3
    grep, cut, and tr are tools specifically designed to do particular jobs. you use grep, for example, instead of writing a bash while-read loop to do a regex or fixed string match against each line of the file. grep pattern file is the way to do it, not while read line ; do [[ $line =~ pattern ]] && echo $line ; done < file
    – cas
    Aug 9, 2021 at 15:19
  • 6
    also, while what terdon said about bash being great for manipulating files is true, it's also true that mostly bash will do that by getting other programs (like mv, cp, tar, and many more including grep, cut, awk, perl, etc etc etc) to do the actual work. bash's number one job is to feed data to other programs via comand-line arguments or by setting up pipes from one to another.
    – cas
    Aug 9, 2021 at 15:22
  • 4
    Exactly yes. The loop is irrelevant, the point is that the shell is not the right tool. Grep, cut, awk, perl, sed, etc are not part of the shell. And, as cas correctly pointed out, neither are mv, cp and their ilk. The point is that the shell is designed to, as cas put it very nicely, orchestrate other programs to process data and do actual work, and isn't good at doing the work itself.
    – terdon
    Aug 9, 2021 at 15:48
5

The other answers are correct, I just wonder if it's difficult to understand the concepts they are conveying. So I want to present you with a metaphor. Hopefully it makes sense:

You can consider the shell script as like a conductor. The conductor points at different musicians to indicate they should start or stop playing, play louder or softer, faster or slower. The conductor themselves doesn't step down into the orchestra to start playing an instrument; neither does the conductor "micro-manage" the musician by telling them how to turn the pages of their music sheets. The conductor trusts the musician to do that work themselves. The conductor just instructs the musicians when to play and how to play.

You shell script is like this. It is orchestrating a lot of other commands. It is not directly manipulating files at all. Even when you use a command like mv or cp, those commands are doing the actual work. They are like the musicians. The shell script says to the mv "musician" — move this here — and it does it. The shell script doesn't move the file itself. It lets the mv command do it.

Also, just like the conductor doesn't tell the musician when to turn the page, the shell script doesn't need to feed a file one line at a time to a command. It can give the whole file to the command and tell it what to do with it. Thus there is no need for a loop. The loop shouldn't be used to micro-manage the command, it should be used to orchestrate multiple commands over multiple files.

Don't forget that "commands" in shell scripts are not like commands in other languages — they are not keywords that form the shell script's language. Rather they are all separate programs that can be run from the shell. cp, for example is a program of its own. That program copies files. It has its own manual and a list of arguments you can pass to it. You can run cp from a shell (outside a script) and all you are doing is calling a program called cp. In the same way you could create your own programs (by creating shell scripts) and call them from within your shell script.

Hope that helped to explain it a bit.

2
  • yes, I guess that one thing to keep in mind, as you mentioned, is that commands are not function of the shell language but actual programs that do things. I guess I knew that (at some level) but, in practice, I've never used these commands as I should have.
    – gabt
    Aug 10, 2021 at 6:26
  • Very poetic analogy. I never thought of cp as its own program, either. Aug 12, 2021 at 16:34
3

I am writing this answer from a pragmatic perspective. I don't think using loops in Bash is necessarily bad.

A big benefit of using scripting languages like Bash is to realize tasks that come to your mind as quickly and easily as possible. Bash enables this by providing you the ability to combine small, efficient, general purpose programs like cat, cut, head, grep, tee, etc. via pipes. One would prefer using a Bash one-liner that makes use of a set of already efficient programs (like grep <something> input-file | cut -c3) rather than writing a program from scratch that does the same job, most of the case even it is slower.

If you roughly know what Bash can offer you, meaning you know sufficient amount of basic unix programs, Bash's piping features and programming language structures, and if it still is convenient to use loops, just do it.

1
  • 5
    Doing that grep pattern file | cut isn't bad. That's using the tools for what they're meant for. But wrapping that command in a shell loop and running grep and cut for each individual line is just silly.
    – ilkkachu
    Aug 10, 2021 at 8:16
-1

There is absolutely nothing wrong with the examples you gave. No, they aren't using processing time efficiently. But that's fine in many circumstances. Only start optimizing once it's actually beneficial. And when you start to have a real need for optimization, you are probably best off dumping bash entirely and using a proper language instead. It's also worth it to learn how to mix bash with your high level language of choice so you can leverage the best of both worlds.

11
  • a long way to go ;-)
    – gabt
    Aug 11, 2021 at 9:01
  • What do you mean? It's really easy to mix, for example Bash+Python works like a charm and is 10 times easier than learning to use all the obscure shell tools that might enable to stay within bash.
    – Nobody
    Aug 11, 2021 at 9:33
  • 1
    @Nobody, there's lots wrong with those examples. Yes, bash's I/O features are slow (by necessity, f/e, read can't buffer input and needs to read one byte at a time so file descriptors are left queued up for other processes to inherit and take over without needing to hand off the buffer content), but if you use bash well you don't need to care if those features are slow; all the work is done by the programs you're calling, not by bash itself. This is an orders-of-magnitude performance difference, more than enough to make the difference between "bash is good enough" and "must rewrite" Aug 11, 2021 at 18:30
  • 1
    @Nobody, ...btw, I'd also argue that awk is a higher-level language itself (and certainly it doesn't have the spec/design limitations that force shell-native IO to be slow). For a reasonable subset of use cases it's just as capable as perl or python. Aug 11, 2021 at 20:17
  • 1
    @gabt, btw, helping folks go from bad bash to good bash is something I do a lot of over at Stack Overflow. Mind, while read isn't itself always bad -- it's slow, but not horrible. Putting var=$(echo "$line" | awk ...) or var=$(echo "$line" | cut ...) inside that loop, on the other, hand, that's horrible. Aug 11, 2021 at 20:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .