7

Is there a way to avoid doing grep two times in the file and just populate the variables in one pass? The file is small so it is not that a big deal I was just wondering if I could do it in one pass

FIRST_NAME=$(grep "$customer_id" customer-info|cut -f5 -d,)
LAST_NAME=$(grep "$customer_id" customer-info|cut -f6 -d,)
  • 3
    How are the variables used later in the code? It may be that they are not needed at all. – Kusalananda Mar 27 '18 at 8:13
  • 4
    @Kusalananda Why would you want to do X? Don't do X. Do Y. – user1717828 Mar 27 '18 at 12:35
13

You could grep once and split twice using shell string substitution:

NAME=$(grep "$customer_id" customer-info | cut -f5,6 -d,)
FIRST_NAME=${NAME%,*}
LAST_NAME=${NAME#*,}

Or, with bash, using process substitution:

IFS=, read FIRST_NAME LAST_NAME < <(grep "$customer_id" customer-info | cut -f5,6 -d,)

read will split input on IFS and assign the first value to FIRST_NAME and the rest to LAST_NAME. Using process substitution and redirection < <(...) allows you to pass the output of grep ... | cut ... to read without using a subshell.

|improve this answer|||||
  • This depends on the fields being sequential. What if it changed to 5 and 7? – Jim Mar 27 '18 at 14:15
  • @Jim Then don't do the cut operation after the grep and use read to break the string by specifying a custom IFS and grab the columns you need from an array. – 111--- Mar 27 '18 at 15:34
  • @Jim it doesn't. Since you can specify whichever fields you want with -f - -fm,n is a list of fields, not a range - this works fine even if the fields aren't next to each other. It does require them to be in order, but if the order is reversed then just swap the variables. – Olorin Mar 27 '18 at 23:19
  • 1
    Nitpick: process substitution does use a subshell -- but it does the read in the main shell thus the variable setting is not lost, while a pipeline something | read var var can do the read in a subshell depending on your shell variant and sometimes options/modes. – dave_thompson_085 Mar 28 '18 at 1:59
  • @dave yes, of the course the point about subshells is only for the read command, not the grep ... | cut pipeline, which is always in a subshell in bash because it's a pipeline. – Olorin Mar 28 '18 at 7:56
4

The simplest would be to get the entire record into a variable and then use cut on that.

RECORD=$(grep "$customer_id" customer-info)
FIRST_NAME=$(echo "$RECORD"|cut -f5 -d,)
LAST_NAME=$(echo "$RECORD"|cut -f6 -d,)

Also personally I would recommend using a more specific regular expression. If your customer IDs are always at the beginning of the line, you can write grep '^'"$customer_id" instead of grep "$customer_id" to require the match to be at the beginning of the line. Otherwise you might pick up records where text matching the customer ID happens to appear elsewhere in the record.

|improve this answer|||||
4

You could use awk in combination with bash read:

read -r FIRST_NAME LAST_NAME <<< $(awk -F, -v cid="$customer_id" '$0~cid{print $5,$6}' customer-info)

-F tells awk to use the comma as field delimiter

-v sets the awk variable cid to the shell variable $customer_id

If the line matches the $customer_id, awk will print the 5th and 6th fields and these will get assigned the variables FIRST_NAME and LAST_NAME.

If first name ($5) ever contains space (example: a,b,c,d,Sarah Jane,Smith) add -v OFS=, to have awk output comma between fields and prefix read with IFS=, to have it split at comma.

Additionally awk can search in only a particular field like '$3~cid{print..}' -- and can match that entire field by '$3~"^"cid"$"{print...}' if that matters for your ids.

|improve this answer|||||
  • Doesn't work as written; edit proposed, with additions since I'm at it. – dave_thompson_085 Mar 28 '18 at 1:47
  • @dave_thompson_085 Your changes are valid one. Thanks. However I do not see any difference between the previous version having /.../ and your version $0~.... Can you also explain what did not work for you? – oliv Mar 28 '18 at 6:36
  • If you just use cid as the pattern, awk does not match the line against the regexp cid -- it only checks if the variable cid is nonempty, whilch is always true, so awk outputs all lines, not the single line you wanted. That's why you need $0 ~ cid -- match the line ($0) against the regexp in cid. – dave_thompson_085 Mar 28 '18 at 22:34
  • When you write a literal regexp /regexp/ {action} the slashes are not part of the regexp, they are special syntax that says there is a regexp. When you put the regexp in a variable if you include a slash that slash is a data character that must match the input data (here, adjacent to the customer_id value) which it almost certainly won't (although conceivably the OP could have data that always has slashes around the customer_id values). – dave_thompson_085 Mar 28 '18 at 22:35
2

Small file, big file. One habit I have is to always eliminate disk IO as much as I can. One way to do this is to push the file into an array. Of course this requires the env $IFS is set appropriately for the file but eliminates the IO.

data=( $(cat customer-info) )

Then you can cherry pick from that...

FIRST_NAME=$(echo "${data[@]}" | tr ' ' '\n' | grep "$customer_id" | cut -f5 -d,)

Another method might be to assign only those two bits desired to an array like so...

data=( $(grep "${customer_id}" customer-info | cut -d, -f5,6) )

|improve this answer|||||
  • 6
    -1 for slurping the entire data set into a Bash variable even though the filter is known beforehand. – David Foerster Mar 27 '18 at 11:44
  • 1
    How do you know the op doesn't require additional data? Assume nothing, eliminate as much overhead io as possible for future operations. Thanks for the concern and opinion however – jas- Mar 27 '18 at 20:14
  • reminds me of the guy who dumped two SQL tables into a variale each, then cross referenced them on a one to many field, then selected the single entry they needed. But I didn't know ASP, so who was I to criticise? – Grump Mar 27 '18 at 23:51
  • Weee, this was the most constructive dialog I have ever engaged in. Someone has poisoned the well. – jas- Mar 28 '18 at 8:25
0

The existing answers all store the output in memory (in a variable) and replay it twice. This is an issue if you want to make a generic wrapper that can take an arbitrarily large input and perform two tasks on it. Instead, the output stream can be duplicated and streamed into two commands.

In my case, the purpose is to filter both the header (first line) and a specific (set of) line(s) in an output stream that may be arbitrarily long. A simple example would be displaying disk space usage:

$ df -h | tee >(head -1 >&2) | grep '/$'
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

Replace df -h with the command you want to use, and replace head -1 and grep '/$' with the two commands you want to apply to it. The output of both will be displayed in your terminal, though it may be that the output of the former command is displayed after the latter.

How does this work?

  • The program tee "[copies] standard input to each [argument], and also to standard output." So it can send output on stdin to both stdout and stderr by using command | tee /dev/stderr.
  • The command >(command2) syntax is replaced by an argument by bash, so command /dev/fd/63 will be executed. When command tries to write to /dev/fd/63, it will end up in the input (stdin) of command2. This is called process substitution (see man bash).
  • Since tee writes both to the argument (we pass a command substitution as an argument) and to stdout, we can just add another pipe and do another command. So now we have command | tee >(command2) | command3.
  • Finally, since command2 will output to stdout, and stdout is piped to command3, we would (in my example) be grepping the header line. That is not what we want: we want to display it. Since we are not piping stderr through, redirecting the output to stderr is an easy way of displaying it in our terminal, i.e. we add >&2, resulting in command | tee >(command2 >&2) | command3.

There is one problem: the output may be in any order. Depending on cosmic rays, we might see either the above or the following:

$ df -h | tee >(head -1 >&2) | grep '/$'
/dev/sda1     202G  145G   57G  72% /
Filesystem    Size  Used Avail Use% Mounted on

A hacky but reliable way to fix this (instead of some over-engineered way that is not hacky) is to add a short sleep to the second command; something like:

$ df -h | tee >(head -1 >&2) | sleep 1; grep '/$'

But wait, this breaks the second command (grep), because now the output is piped from tee to sleep and grep will be waiting for input indefinitely. To fix that, we add a subshell:

$ df -h | tee >(head -1 >&2) | (sleep 0.01; grep '/$')
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

Now the output is not redirected to grep but to our subshell. Since sleep does not read from it (it does not consume the stream), it is still available for grep to read. Now it works reliably so long as head outputs within 0.01 seconds (plus a bit of overhead on the grep side), which is a fair bet on a modern system and short enough not to be noticeable to the user.

Since I wanted to make something that takes both the header and the output of some command, we can generalize this to:

function grabheader {
    tee >(head -1 >&2)
}

Since the tee command in the function will just read from stdin and output to stdout, this does the same as our earlier out-of-order command when you use it as df -h | grabheader | grep '/$'. But since we want it to be in-order, we need to delay sending it down the stdout:

function grabheader {
    tee >(head -1 >&2) | (sleep 0.01; cat)
}

cat here just makes sure that whatever is passed to the stdin will make its way onto the stdout again. By passing no arguments and adding no redirections, it will do just that. Usage:

$ df -h | grabheader | grep '/$'
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

Of course, in the particular case of df, this can be done much simpler:

$ df -h /
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda1     202G  145G   57G  72% /

But now we have a general way of doing this with any command.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.