The power of shell pipeline is so great that sometimes fails me.


Just as an example, the pipeline

echo abc > file.txt
cat file.txt | sed 's/a/1/' > file.txt

gives me an empty file.txt. Realizing that the shell probably calls > first, I made a change:

echo abc > file.txt
{cat file.txt | sed 's/a/1/'} > file.txt

Again it surprises me by another empty file file.txt. An ugly way that finally works is

echo abc > file.txt
echo $(cat file.txt | sed 's/a/1') > file.txt

which forces the shell to run a subshell first, and then redirect.


While I'm aware of better practice of sed, which allows you to get rid of echo, cat, grep.. etc, what I am curious about here is to learn shell's grammar completely. This questions is not about how to fix the particular problem above.

Q1(EDIT: off-topic) Is there a good resource for me to learn the grammar?

I'm afraid that different shells could have different grammars, so

Q2 Can I make shell verbose, and see clearly what it is doing every completely every single time I run a command?

Q3(EDIT: off-topic) Any other advice on good practices? Thank you!

  • unix.stackexchange.com/a/443154 may be helpful – Quasímodo May 13 '20 at 19:16
  • Q1 & Q3 are off-topic for U&L; please edit your question down to Q2 only. Thank you! – Jeff Schaller May 13 '20 at 19:23
  • 1
    The easiest way to enable "verbose" is to turn on the trace flag: bash -x script.sh or in an interactive shell set -x – glenn jackman May 13 '20 at 20:20
  • Good resources to learn the grammar: not really. In my experience, the best way is to practice practice practice, ask questions, and follow the shell tag here and on stackoverflow. – glenn jackman May 13 '20 at 20:22
  • Thank you.. @SergiyKolodyazhnyy but that was just an example to illustrate my situation: I not only want to solve this problem, but also want to avoid similar problems in the future. – Student May 14 '20 at 3:38

You have successfully found one of the things you should not do :-) Never redirect to the file you are working on!

A1: a good resource to learn shell grammar would be the absolute bash scripting guide, IMO

A2: For bash-scripts, you can use set +x for more verbose output, but I don't know how to achieve the same at a 'run things at the prompt'-level.

A3: Add [solved] to your search-terms. Finds you the solution to your problem instead of more of the problem you already know.

  • .. Having set +x, it still doesn't hint me why there's a problem. I guess I should just make friend with shell(s), and know all of its (their) perks. – Student May 14 '20 at 0:53

Consider 3.1.1 Shell Operation, particularly the order in which things are done:

  • redirections are processed before the command is actually executed, but
  • expansions are processed before redirections.

This means that for cat file > file, the output redirection (which truncates the file) occurs before cat is spawned, and cat now has an empty file to work with.

But echo "$(cat file)" > file does what you expect because the Command Substitution is a Shell Expansion, and that happens before redirections.

The typical advice is to do

cat file > tmpfile && mv tmpfile file

You can use mktemp here.

Or install the moreutils package and use sponge

cat file | sponge file

Although to address the specific command you're using, replace

cat file.txt | sed 's/a/1/' > file.txt

with (assuming GNU sed)

sed -i 's/a/1/' file.txt

One way of solving the problem is learning exactly what order redirections, expansions, subshells etc happen so that you can catch this type of error before it happens.

I do not recommend this. You will still mess up occasionally and the cost can be lost data.

It is better to not trust your ability to keep all this straight and write more idiot-proof code. We are all idiots sometimes.


cp file.txt file.txt.old
cat file.txt.old | sed 's/a/1/' > file.txt

This is simpler in that there is no dependencies on order of events, other than line 1 happening before line 2.

As an added benefit it guards against other errors, when you shouldn't have run this script at all.

The cost of this practice is that you will have .old files laying around everywhere. This is an insurance premium you will be glad you paid the day you need it. Disk is cheap.

  • You don't recommend learning the order of evaluation? – Kusalananda May 14 '20 at 8:11
  • @Kusalananda Well, mostly I recommend not depending on that knowledge. But feel free to learn it if you want to. – Stig Hemmer May 14 '20 at 8:20
  • Well, I guess @StigHemmer just recommends to backup always. Thanks for reminding :) – Student May 14 '20 at 11:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.