Can someone clarify this Bash script

Question

I'm learning something about Bash scripting, and I came across this example:

The task is to write a Bash script that reads a file, given with the first argument, pulls out those records that have the same ID which is given with the second argument and live in the place given with the third argument.

So, the file is a list of users, with info about one user per line, and it looks something like this (ID, name, father's name, location, phone)

43  John  Mike  Smith  Boston  +3 685 123456

After the "filtering", I have to write the found data to standard output, with the following modifications : first letter of the location should be concatenated with the ID, and then the name should only contain the first letter of the father's name and a phone without any prefixes.

The output for the given example would look like this:

B43 John M Smith 123456

The solution is:

#!/bin/bash
cat $1|grep "^$2[0-9]*.*$3  +[0-9]*\ [0-9]*\ [0-9]*"|
sed "s/\([0-9]*\)\t\(.*\)\t\(.\).*\t\(.*\)\t\(.\).*\t+[0-9]*\ [0-9]*\
\([0-9]*\)/\5\1 \2 \3 \4 \6/"

I don't understand what's the point of | vertical lines - I understand they are pipes, and the output data from one 'query' serves as a input data for another 'query'. By query I mean a shell command.

I get the part with the grep command.

I don't get the sed command. How exactly does this work? How does it 'know' to put the first letter of the location at the beginning of the line?

Answer 1

Line-by-line and pipe-by-pipe explanation:

#!/bin/bash

This is a so-called shebang - basically tells it to run this script with the program /bin/bash.

cat $1

$1 is the first script argument. cat $1 is going to output the contents of the file supplied as first script argument to stdout. However, since there's a pipe after this, the stdout will be piped to stdin of the next command in the pipeline, grep in this case.

grep "^$2[0-9]*.*$3  +[0-9]*\ [0-9]*\ [0-9]*"

This will read the stdin (what cat $1 output above). You can read about grep more e.g. here:

• man page: http://unixhelp.ed.ac.uk/CGI/man-cgi?grep
• Some examples: http://www.cyberciti.biz/faq/howto-use-grep-command-in-linux-unix/

The above will filter the lines supplied to it by regex. The regex:

^$2[0-9]*.*$3  +[0-9]*\ [0-9]*\ [0-9]*

basically says that we want lines that:

• Start with the second script argument ($2 above),
• Then have zero or more digits ([0-9]*)
• Followed by zero or more occurrences of (almost) any character
• Followed by the third script argument ($3)
• Followed by two or more spaces (+ - note that there are two spaces here)
• Followed by zero or more digits ([0-9]*)
• Followed by a space (\)
• Followed by zero or more digits ([0-9]*)
• Followed by a space (\)
• Followed by zero or more digits ([0-9]*)

All lines that match the above will be output to stdout. Again, stdout is piped to the stdin of the next command, sed in this case.

sed "s/\([0-9]*\)\t\(.*\)\t\(.\).*\t\(.*\)\t\(.\).*\t+[0-9]*\ [0-9]*\
\([0-9]*\)/\5\1 \2 \3 \4 \6/"

You can read about sed more e.g. here:

• man page: http://unixhelp.ed.ac.uk/CGI/man-cgi?sed
• Some examples: http://www.grymoire.com/unix/sed.html

The above basically says, for each line:

• Substitute (s/)
• (A) This: \([0-9]*\)\t\(.*\)\t\(.\).*\t\(.*\)\t\(.\).*\t+[0-9]*\ [0-9]*\([0-9]*\)
• (B) With this: \5\1 \2 \3 \4 \6

The part marked with (A) above is the regexp again, similar to what grep used. What it says is a bit more regular. Note that it has a structure along the lines X\tY\tZ\t.... What this essentially says to sed is - match a line that has tab characters (this is what \t means) and some things (X, Y, Z) in between. Those things in the above can be grouped in two ways:

• Expressions such as \([0-9]*\) are so-called regex capturing groups. They are basically delimited by parentheses, except that sed is a bit older than what is the usual way to do in regexps today. E.g. if you used a regex tool such as http://regexpal.com/, you'd put ([0-9]*) instead. sed needs these to be escaped to signify groups - otherwise it would think it needs to match the actual parentheses. It can be instructed to do the reverse by supplying -r command line option
• Expressions outside of the escaped parentheses (e.g. the part \t+[0-9]*)

The capturing groups is what allows sed to do what you are asking for. Note the (B) part of the sed command. It says this:

\5\1 \2 \3 \4 \6

This is actually a nice way of saying - replace what I matched in this line the 5th capturing group, then 1st group, then a space, then a 2nd group, etc.

In order to make it clearer, here's a sample command for you to try:

echo abc|sed 's/\(.\)\(.\)\(.\)/\3\2\1/'

or if you want in a extended regex format which is easier to read:

echo abc|sed -r 's/(.)(.)(.)/\3\2\1/'

Run it and see what this outputs - noting that echo outputs three characters in the line and that for sed part . matches (almost) any character, it should be clear what and how it applies to your situation. I suggest you to play with some examples of sed substitution on the Net - that should be the best way to clear things up.