2

I am trying to find a better way to complete the following grep/awk query. The below is a trivial example of the question.

I have gotten to this point with my regex:

grep -Po ^(?:[^,]+,\s?){7}(Want|Need) | awk -F ',' 'NR>=2{print $8}' | sort | uniq -c

My CSV file looks as follows:

1896,Ranger,2021,State,Postcode,Surname,Industry,Want,Turbo,Good
1896,Ranger,2021,State,Postcode,Surname,Industry,Selling,Turbo,Good
1896,Ranger,2021,State,Postcode,Surname,Industry,Need,Turbo,Good

The above works to print the entire row with grep:

1896,Ranger,2021,State,Postcode,Surname,Industry,Want
1896,Ranger,2021,State,Postcode,Surname,Industry,Need

and I can then count values from column 8. My question is how do I write the grep/regex query to only return the group I have selected with regex.

eg:

Want
Need

The reason for this post is to purely understand a better way to use regex here. I know of other ways to do this.

1

4 Answers 4

3

It sounds like you're looking for the PCRE \K assertion. From perlre:

There is a special form of this construct, called \K (available since Perl 5.10.0), which causes the regex engine to "keep" everything it had matched prior to the \K and not include it in $&.

So

$ grep -Po '^(?:[^,]+,\s?){7}\K(Want|Need)' file.csv
Want
Need

More generally, this kind of thing is done with a lookbehind assertion - however Perl doesn't support variable length lookbehinds, and neither does grep -P:

$ grep -Po '^(?<=(?:[^,]+,\s?){7})(Want|Need)' file.csv
grep: lookbehind assertion is not fixed length

See also Lookahead and Lookbehind Zero-Length Assertions

4
  • \K did the trick. Thanks
    – malkier11
    Commented Aug 26, 2021 at 7:28
  • Note that the PCRE regexp specification is documented in the pcrepattern man page. There are some differences compared to perl regexps including for \K (though none that applies here). \K was added in PCRE 7.2. Commented Aug 26, 2021 at 10:38
  • can this be used with sed to replace the values? eg: want->value_1, need->value_2
    – malkier11
    Commented Aug 26, 2021 at 10:59
  • @malkier11 any time you're considering using sed+grep you should be using awk instead.
    – Ed Morton
    Commented Aug 26, 2021 at 21:24
3

Note that -P is a non-standard (and optional and long considered experimental) option of the GNU implementation of grep, with which it uses libpcre (a standalone implementation of perl's regex) to do the matching¹

libpcre does come with its own grep command as example code (pcregrep), though it has now evolved into a full-fledged grep implementation, that for instance can be found in its own package on a few GNU/Linux distributions.

pcregrep has extended GNU grep's -o non-standard option to take an optional numeric argument to output the corresponding capture group:

So here:

pcregrep -o1 '^(?:[^,]+,\s?){7}(Want|Need)'

Or you could use the real thing, which would also have the advantage of working even on systems that have neither GNU grep (or where GNU grep has been built without PCRE support) nor pcregrep:

perl -lne 'print $1 if /^(?:[^,]+,\s?){7}(Want|Need)/'

Note however that perl, by default doesn't decode the input as per the locale's text encoding like GNU grep does. In that specific case, where the text you're matching uses only characters from the portable character set, it's probably rather and advantage as it will still work even if the input's encoding differs from the locale.

If you wanted perl to decode text on input (and encode on output) as per the locale's encoding, you could add -Mopen=locale.


In your case though, there's not much that warrants perl regexps. All the perl operators you're using there have standard ERE operator equivalents (even BRE except for alternation).

  • (?:...): it's just perl/ERE (...) or BRE \(...\) without capturing.
  • +: same in ERE, \{1,\} in BRE
  • ?: same in ERE, \{0,1\} in ERE
  • {7}: same in ERE, \{7\} in BRE
  • (Want|Need): same in ERE (though with slight differences in behaviour when it comes to choosing sides of the alternation).
  • \s: [[:space:]] in both BRE and ERE
  • ^, [^,]: same in BRE or ERE

sed is the tool to extract matching parts in a pattern (while grep, named after ed's g/re/p command is to print the lines that match a regular expression). Standard sed uses BREs, but most sed implementations support -E to switch to ERE (and that will be added to the next version of the standard).

So here, as an equivalent of the perl command above, you could also do portably:

LC_ALL=C sed -nE 's/^([^,]+,[[:space:]]?){7}(Want|Need).*$/\2/p'

Or without -E:

LC_ALL=C sed -n 's/^\([^,]\{1,\},[[:space:]]\{0,1\}\)\{7\}\(Want\).*$/\2/p; t
                 s/^\([^,]\{1,\},[[:space:]]\{0,1\}\)\{7\}\(Need\).*$/\2/p'

Or to replace those Want or Need with something else:

LC_ALL=C sed -E 's/^(([^,]+,[[:space:]]?){7})(Want|Need)/\1Desire/'
LC_ALL=C sed 's/^\(\([^,]\{1,\},[[:space:]]\{0,1\}\)\{7\}\)Want/\1Desire/; t
              s/^\(\([^,]\{1,\},[[:space:]]\{0,1\}\)\{7\}\)Need/\1Desire/

¹ since then, other implementations have added their own -P option to use perl-like regular expressions, not always using libpcre like ast-open's grep (which does support look-around assertions but not \K)

3
  • Thanks, are you able to give me an idea of how I could replace the values using sed. eg want->value_1, need->value_2. I understand how to do it with awk, more for my own curiosity.
    – malkier11
    Commented Aug 26, 2021 at 11:24
  • @malkier11 see edit. Commented Aug 26, 2021 at 11:47
  • This is awesome, thanks so much for the detailed response.
    – malkier11
    Commented Aug 28, 2021 at 3:18
0

You're already using awk, so you don't need grep here. You don't need sort or uniq -c either. For example:

$ awk -v search=Want -F, '$8 ~ search { count[$8]++ };
    END { for (f in count) { printf "%5i\t%s\n", count[f], f}}' input.csv 
    1   Want

$ awk -v search='Want|Need' -F, '$8 ~ search { count[$8]++ };
    END { for (f in count) { printf "%5i\t%s\n", count[f], f}}' input.csv 
    1   Want
    1   Need

Or if you want it to print the matching lines too:

$ awk -v search='Want|Need' -F, '$8 ~ search { count[$8]++ ; print };
    END { for (f in count) { printf "%5i\t%s\n", count[f], f}}' input.csv 
1896,Ranger,2021,State,Postcode,Surname,Industry,Want,Turbo,Good
1896,Ranger,2021,State,Postcode,Surname,Industry,Need,Turbo,Good
    1   Want
    1   Need

You can add case-insensitivity with GNU awk by adding -v IGNORECASE=1 to the command line, and even add fancy feature like exact matches if you want:

$ awk -v search='want' -v exact=1 -v IGNORECASE=1 -F, '
    BEGIN {if (exact == 1) search = "^(" search ")$"};
    $8 ~ search { count[$8]++ ; print };
    END { for (f in count) { printf "%5i\t%s\n", count[f], f}}' input.csv 
1896,Ranger,2021,State,Postcode,Surname,Industry,Want,Turbo,Good
    1   Want

The following won't produce any output because while ant is in Want, it is not an exact match for the entire field 8:

$ awk -v search='ant' -v exact=1 -v IGNORECASE=1 -F, '
    BEGIN {if (exact == 1) search = "^(" search ")$"};
    $8 ~ search { count[$8]++ ; print };
    END { for (f in count) { printf "%5i\t%s\n", count[f], f}}' input.csv 

Note: there are obviously better ways of doing command-line option processing (e.g. with a getopt function or by writing a shell script wrapper to use the sh/bash built-in getopt), but using awk's -v option to set variables in awk from outside the script is easy and convenient for simple tasks like this.

BTW, awk also allows variables to be asssigned without using -v by adding them to the command line after the script itself (awk will interpret any argument of the form x=y as setting variable x to value y. Unfortunately, this makes it difficult to use filenames with an = in them - maybe impossible, I can't remember if I ever found a solution to this other than "don't do that, then").

Unlike when using -v, though, these variables are NOT available in a BEGIN {} statement. e.g. the following will match ant even though we're setting exact=1:

$ awk -F, 'BEGIN {if (exact == 1) search = "^(" search ")$"};
           $8 ~ search { count[$8]++ ; print };
           END { for (f in count) { printf "%5i\t%s\n", count[f], f}}' \
    search=ant IGNORECASE=1 exact=1 input.csv 
1896,Ranger,2021,State,Postcode,Surname,Industry,Want,Turbo,Good
    1   Want

From GNU awk's man page:

If a filename on the command line has the form var=val it is treated as a variable assignment. The variable var will be assigned the value val. (This happens after any BEGIN rule(s) have been run.)

Command line variable assignment is most useful for dynamically assigning values to the variables AWK uses to control how input is broken into fields and records. It is also useful for controlling state if multiple passes are needed over a single data file.

IMO, it's best to think of that as a legacy feature for compatibility with old awk scripts and to just use -v.

-v var=val
--assign var=val

Assign the value val to the variable var, before execution of the program begins. Such variable values are available to the BEGIN rule of an AWK program.

(bold emphasis on "after" and "are" in the quotes above were added by me)

4
  • 1
    Assigning variables in the arg list isn't just legacy, its commonly used to assign values between files (e.g. to work on a file that's ,-separated and one thats tab-separated in the same script you can use awk 'script' FS=',' file1 FS='\t' file2, or to identify the 2nd file rather than testing NR==FNR in the script you can use awk 'script' file1 f=1 file2). To use a file name that contains = you just have to specify the path to it, e.g. x=7 is a variable assignment but ./x=7 is a file name. See stackoverflow.com/q/19075671/1745001 for more info on initializing awk variables
    – Ed Morton
    Commented Aug 26, 2021 at 14:15
  • Thank you for your opinion. I'll give it all the consideration it deserves.
    – cas
    Commented Aug 26, 2021 at 16:08
  • You're welcome. Nothing in my comment is an opinion though, it's all facts. If you think something I said is not a fact, please do let me know what and I'll clarify and/or provide examples and additional documentation.
    – Ed Morton
    Commented Aug 26, 2021 at 20:58
  • Feel free to keep telling yourself that.
    – cas
    Commented Aug 27, 2021 at 1:31
0

Assuming that if one of the strings you're searching for doesn't appear in the input you want to see it printed with a count of 0 rather than not printed at all, the robust, portable, efficient, concise way to do this is simply:

$ awk -F',' -v tgts='Want,Need' '
    { cnt[$8]++ }
    END { split(tgts,t); for (i in t) print t[i], cnt[t[i]]+0 }
' file
Want 1
Need 1

so it's hard to figure out where a regexp would come into play here. Maybe the following:

$ awk -F',' -v tgts='Want|Need' '
    $8 ~ ("^"tgts"$") { cnt[$8]++ }
    END { split(tgts,t,/[|]/); for (i in t) print t[i], cnt[t[i]]+0 }
' file
Want 1
Need 1

or:

$ awk -F',' -v tgts='Want|Need' '
    $0 ~ ("([^,]*,){7}"tgts"(,|$)") { cnt[$8]++ }
    END { split(tgts,t,/[|]/); for (i in t) print t[i], cnt[t[i]]+0 }
' file
Want 1
Need 1

but the regexps are just complicating the scripts and making them more fragile (the ones with regexps would fail if the strings you wanted to find contained regexp metachars like . or * while the first script would continue to work) and adding no value unless you have billions of unique $8 values in your input.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .