How could I grep twice in one pass?

Question

Is there a way to avoid doing grep two times in the file and just populate the variables in one pass? The file is small so it is not that a big deal I was just wondering if I could do it in one pass

FIRST_NAME=$(grep "$customer_id" customer-info|cut -f5 -d,)
LAST_NAME=$(grep "$customer_id" customer-info|cut -f6 -d,)

Answer 1

You could grep once and split twice using shell string substitution:

NAME=$(grep "$customer_id" customer-info | cut -f5,6 -d,)
FIRST_NAME=${NAME%,*}
LAST_NAME=${NAME#*,}

Or, with bash, using process substitution:

IFS=, read FIRST_NAME LAST_NAME < <(grep "$customer_id" customer-info | cut -f5,6 -d,)

read will split input on IFS and assign the first value to FIRST_NAME and the rest to LAST_NAME. Using process substitution and redirection < <(...) allows you to pass the output of grep ... | cut ... to read without using a subshell.

Answer 2

The simplest would be to get the entire record into a variable and then use cut on that.

RECORD=$(grep "$customer_id" customer-info)
FIRST_NAME=$(echo "$RECORD"|cut -f5 -d,)
LAST_NAME=$(echo "$RECORD"|cut -f6 -d,)

Also personally I would recommend using a more specific regular expression. If your customer IDs are always at the beginning of the line, you can write grep '^'"$customer_id" instead of grep "$customer_id" to require the match to be at the beginning of the line. Otherwise you might pick up records where text matching the customer ID happens to appear elsewhere in the record.

Answer 3

You could use awk in combination with bash read:

read -r FIRST_NAME LAST_NAME <<< $(awk -F, -v cid="$customer_id" '$0~cid{print $5,$6}' customer-info)

-F tells awk to use the comma as field delimiter

-v sets the awk variable cid to the shell variable $customer_id

If the line matches the $customer_id, awk will print the 5th and 6th fields and these will get assigned the variables FIRST_NAME and LAST_NAME.

If first name ($5) ever contains space (example: a,b,c,d,Sarah Jane,Smith) add -v OFS=, to have awk output comma between fields and prefix read with IFS=, to have it split at comma.

Additionally awk can search in only a particular field like '$3~cid{print..}' -- and can match that entire field by '$3~"^"cid"$"{print...}' if that matters for your ids.

Answer 4

Small file, big file. One habit I have is to always eliminate disk IO as much as I can. One way to do this is to push the file into an array. Of course this requires the env $IFS is set appropriately for the file but eliminates the IO.

data=( $(cat customer-info) )

Then you can cherry pick from that...

FIRST_NAME=$(echo "${data[@]}" | tr ' ' '\n' | grep "$customer_id" | cut -f5 -d,)

Another method might be to assign only those two bits desired to an array like so...

data=( $(grep "${customer_id}" customer-info | cut -d, -f5,6) )

Answer 5

The existing answers all store the output in memory (in a variable) and replay it twice. This is an issue if you want to make a generic wrapper that can take an arbitrarily large input and perform two tasks on it. Instead, the output stream can be duplicated and streamed into two commands.

In my case, the purpose is to filter both the header (first line) and a specific (set of) line(s) in an output stream that may be arbitrarily long. A simple example would be displaying disk space usage:

$ df -h | tee >(head -1 >&2) | grep '/$'
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

Replace df -h with the command you want to use, and replace head -1 and grep '/$' with the two commands you want to apply to it. The output of both will be displayed in your terminal, though it may be that the output of the former command is displayed after the latter.

How does this work?

• The program tee "[copies] standard input to each [argument], and also to standard output." So it can send output on stdin to both stdout and stderr by using command | tee /dev/stderr.
• The command >(command2) syntax is replaced by an argument by bash, so command /dev/fd/63 will be executed. When command tries to write to /dev/fd/63, it will end up in the input (stdin) of command2. This is called process substitution (see man bash).
• Since tee writes both to the argument (we pass a command substitution as an argument) and to stdout, we can just add another pipe and do another command. So now we have command | tee >(command2) | command3.
• Finally, since command2 will output to stdout, and stdout is piped to command3, we would (in my example) be grepping the header line. That is not what we want: we want to display it. Since we are not piping stderr through, redirecting the output to stderr is an easy way of displaying it in our terminal, i.e. we add >&2, resulting in command | tee >(command2 >&2) | command3.

There is one problem: the output may be in any order. Depending on cosmic rays, we might see either the above or the following:

$ df -h | tee >(head -1 >&2) | grep '/$'
/dev/sda1     202G  145G   57G  72% /
Filesystem    Size  Used Avail Use% Mounted on

A hacky but reliable way to fix this (instead of some over-engineered way that is not hacky) is to add a short sleep to the second command; something like:

$ df -h | tee >(head -1 >&2) | sleep 1; grep '/$'

But wait, this breaks the second command (grep), because now the output is piped from tee to sleep and grep will be waiting for input indefinitely. To fix that, we add a subshell:

$ df -h | tee >(head -1 >&2) | (sleep 0.01; grep '/$')
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

Now the output is not redirected to grep but to our subshell. Since sleep does not read from it (it does not consume the stream), it is still available for grep to read. Now it works reliably so long as head outputs within 0.01 seconds (plus a bit of overhead on the grep side), which is a fair bet on a modern system and short enough not to be noticeable to the user.

Since I wanted to make something that takes both the header and the output of some command, we can generalize this to:

function grabheader {
    tee >(head -1 >&2)
}

Since the tee command in the function will just read from stdin and output to stdout, this does the same as our earlier out-of-order command when you use it as df -h | grabheader | grep '/$'. But since we want it to be in-order, we need to delay sending it down the stdout:

function grabheader {
    tee >(head -1 >&2) | (sleep 0.01; cat)
}

cat here just makes sure that whatever is passed to the stdin will make its way onto the stdout again. By passing no arguments and adding no redirections, it will do just that. Usage:

$ df -h | grabheader | grep '/$'
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

---

Of course, in the particular case of df, this can be done much simpler:

$ df -h /
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

But now we have a general way of doing this with any command.