0

TL;DR: Is there a way to print the inputs and outputs of every step of a bash shell script, including the steps between pipe commands?

Hopefully this is relatable: I wrote a script a while ago and now want to edit it, but I don't understand how it works anymore. I'll use this script as an example:

#! /usr/bin/bash
set -ev

TREE_IGNORE_PATTERN='node_modules|package-lock.json|.gitignore|.git|dist'

echo '### Project Structure'
echo '```'
tree --noreport -F -a -I $TREE_IGNORE_PATTERN | tr -d '*'
echo '```'
echo ""

tree --noreport -nFif -I $TREE_IGNORE_PATTERN | awk '$0 !~ /\/$/ && $0 !~ /^\.$/' | tr -d '*' | awk '{print "" ; print "### " $0 ; print "```" ; system("cat " $0 ) ; print "" ; print "```" ; print "" }'

As you can see, I set -v at the top. This helps because it tells me what's actually executing, but it's not enough to understand how each step of the pipes work. What would really help is seeing the input transform step-by-step. Since this is so informative, I normally go about it manually:

  1. I forget how tree --noreport -nFif -I $TREE_IGNORE_PATTERN | awk '$0 !~ /\/$/ && $0 !~ /^\.$/' | tr -d '*' | awk '{print "" ; print "### " $0 ; print "```" ; system("cat " $0 ) ; print "" ; print "```" ; print "" }' works, so I would copy it.
  2. I paste it into my commandline.
  3. Since it depends on a variable named $TREE_IGNORE_PATTERN:
    1. I copy the expanded value of that variable.
    2. I replace that variable on the command line with the expanded value.
    3. Now the command looks like this: tree --noreport -nFif -I 'node_modules|package-lock.json|.gitignore|.git|dist' | ....
  4. I press enter on the command line to run step 3.3. This is to save the command in my history so I can get back to it by pressing up.
  5. I press up.
  6. I edit the command line by removing all piped commands, leaving me with tree --noreport -nFif -I 'node_modules|package-lock.json|.gitignore|.git|dist'.
  7. I run that to get a sense of the output.
  8. I press up.
  9. I edit the command line by removing all piped commands but 1, leaving me with tree --noreport -nFif -I 'node_modules|package-lock.json|.gitignore|.git|dist' | awk '$0 !~ /\/$/ && $0 !~ /^\.$/'.
  10. I run that to get a sense of the output.
  11. I repeat until I'm running the step 3.3 on the command line again. Now I can see every step of the input's transformation.

This is a lot of manual work. Is there an automated way of seeing the input and output of each pipe command in a bash shell script?

If the answer is, "No, but zsh can," I'd accept that as an answer (but only if bash can't). I'd also accept, "No, but if you wrote/commented your code like this, it would be easier to understand when you revisit it," but again, only if bash and zsh don't have a way to automate this.

Improve this question
4
  • off the top of my head, doesn't set -v just print the input lines as-is, it doesn't really show what the shell does. On the other hand, set -x would show the commands as it runs them, though not the exact source command line that produces them.
    – ilkkachu
    Commented Dec 5, 2023 at 8:08
  • @ilkkachu The man say page seems to say that, but when I used it, the only difference was when say the commands are shown. -x say shows when they are run, I can't remember when -v does, unfortunately. The explanation was elsewhere on this site.
    – Daniel Kaplan
    Commented Dec 6, 2023 at 22:23
  • well, my experience seems to match what the manual says, e.g. bash -vc 'for x in 1 2 3; do echo $x; done' prints that whole line, once, and nothing more, while the same with -x instead prints each of echo 1, echo 2, echo 3 (plus a line for the for itself) while the loop runs. The output is different, but yes, of course it shows in the timing too. It's even more evident if you add a sleep 1 to the loop.
    – ilkkachu
    Commented Dec 7, 2023 at 7:21
  • @ilkkachu oh I misunderstood what you/man meant by "input lines". You're right. FWIW, I use -x instead of -v most of the time.
    – Daniel Kaplan
    Commented Dec 7, 2023 at 9:28

1 Answer 1

Reset to default
2

If you want them to not be confused and you only need to debug one pipeline at a time you could use tee to pipe a copy of each output to a file. In your case:

tree --noreport -nFif -I $TREE_IGNORE_PATTERN | awk '$0 !~ /\/$/ && $0 !~ /^\.$/' | tr -d '*' | awk '{print "" ; print "### " $0 ; print "```" ; system("cat " $0 ) ; print "" ; print "```" ; print "" }'

would become:

tree --noreport -nFif -I $TREE_IGNORE_PATTERN | tee out1.log | awk '$0 !~ /\/$/ && $0 !~ /^\.$/' | tee out2.log | tr -d '*' | tee out3.log | awk '{print "" ; print "### " $0 ; print "```" ; system("cat " $0 ) ; print "" ; print "```" ; print "" }'
Improve this answer
2
  • Hm. In some ways this is more convenient, in others it is less. I can skip the command line, but now I've got to read and clean up those files.
    – Daniel Kaplan
    Commented Dec 5, 2023 at 0:15
  • 1
    Typically, I would maintain both the debug and plain versions of the pipeline in the script, and select which I run using a --debug option. Also, put in a comment block for each intermediate format. Have all the debug file names with a date/time and a common suffix, so you can compare old/new for each pipe, and clean them up with a wildcard on either. If you know you need to maintain and retest a complex script, plan for it up front.
    – Paul_Pedant
    Commented Dec 5, 2023 at 9:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .