2

I feel like this is a really easy question, and when I Google, I find lots of answers for part of the problem, but when I try to put them together, it doesn't work and I can't figure out why.

Here's the scenario:

  1. I have a file with a lot of text in it.
  2. One of those lines matches this pattern: foo = 1700;
  3. I want to extract 1700
  4. I want to save it into a bash script variable so I can refer to it later in the script.

I cannot get past step 3. Here's what I've tried:

sed -nE 's/^foo = //p' file | sed -nE 's/;//p'

This prints out:

1700

Great, but what if I need to trim white space or something? If I can't use */+, I wouldn't know how to do that. I learned that you can't use */+ in a substitution on another answer, so I can't figure out how to do this. I looked into the man page of grep, and I didn't see any option for groups when I search for that word. I think I know how to solve this in awk, but I've always found its regex functions to be a little clunky and for the commandline scripts to require too many escapes, so ideally that's not the only way to solve this.

4
  • Why do you want to do this with shell & command-line apps? To me, it seems far easier to write a few lines of (f)lex code. – jamesqf Jun 12 at 16:08
  • What is this answer that told you you can't use * or + in a substitution? That sounds quite strange. – David Z Jun 13 at 0:45
  • This seems to be a similar question: askubuntu.com/questions/184204/… – RonJohn Jun 13 at 0:48
  • @RonJohn It is, but I'd say mine has a larger scope because it's not focused on grep – Daniel Kaplan Jun 13 at 2:10
7
  1. To start with, here's how to capture the numeric value:

    $ echo 'foo = 1700;' | sed -n -e 's/^foo = \([0-9]\+\).*/\1/p'
    1700
    

    That's using sed's default Basic Regular Expressions (BRE). You can also use Extended Regular Expressions (ERE) with sed's -E option:

    echo 'foo = 1700;' | sed -n -E -e 's/^foo = ([0-9]+).*/\1/p'
    1700
    

    The sub-expression [0-9]+ inside the parentheses (...) captures one-or-more digits. This is called a "capture group" and is used in the replacement with the \1 (which is the first capture group - if there are multiple capture groups, they can be used as \1, \2, \3, etc).

    In this case, the sed script tries to replace the entire line with just the \1 capture group and if that succeeded, print the modified line.

  2. Next, you want to get sed's output into a variable. You do that with with command substitution. e.g.

    $ myvar=$(echo 'foo = 1700;' | sed -n -E -e 's/^foo = ([0-9]+).*/\1/p')
    $ echo $myvar
    1700
    
  3. To use this in your script, just use your file as an argument to sed instead of piping echo ... into it.

    myvar=$(sed -n -E -e 's/^foo = ([0-9]+).*/\1/p' file)
    
  4. To trim white space, or to cope with lines that might have optional leading whitespace, or optional whitespace around the =, etc:

    myvar=$(sed -n -E -e 's/^[[:space:]]*foo[[:space:]]*=[[:space:]]*([0-9]+).*/\1/p' file)
    

    Note that some versions of sed (GNU sed, at least. maybe others) understand perl's \s, so you can shorten that to:

     myvar=$(sed -n -E -e 's/^\s*foo\s*=\s*([0-9]+).*/\1/p' file)
    
9
  • wait so does sed not have \d? – Daniel Kaplan Jun 12 at 6:50
  • 2
    okay that explains the only problem I was having the whole time. Thanks! – Daniel Kaplan Jun 12 at 6:51
  • 2
    @DanielKaplan there was no mention of that anywhere in your question – roaima Jun 12 at 7:03
  • 1
    @Kusalananda GNU grep -P uses libpcre and not some GNU implementation of it, so I expect it will behave identically to anything else using PCRE. Whether that's identical to perl's behaviour, I don't know. perl has good unicode support. man pcre says The current implementation of PCRE corresponds approximately with Perl 5.12, including support for UTF-8/16/32 encoded strings and Unicode general category properties. However, UTF-8/16/32 and Unicode support has to be explicitly enabled; it is not the default. The Unicode tables correspond to Unicode release 6.3.0. – cas Jun 12 at 7:43
  • 1
    @roaima true, admittedly, I didn't want to post every failed attempt I made 1) because I found something that kind of worked but felt clunky, 2) somehow i got it in my head that sed's s/... can't use */+ (I must have misunderstood a SO answer for one of the parts I was researching before I posted this). In my head that misinterpretation was an invariant. At that point, bringing that detail up seemed irrelevant, like bringing up which hand I used for the mouse as I tried to get this to work. TL;DR: When you assume, you make an... – Daniel Kaplan Jun 12 at 22:53
3

For completeness, with grep implementations that support -o and perl-like regular expressions with -P, you can do:

grep -Po 'foo\s*=\s*\K\d+'

Where:

  • \s matches any whitespace character
  • * 0 or more of the preceding atom. So \s* for instance matches 0 or more whitespace characters.
  • \d matches a decimal digit (generally the same as [0123456789], though not [0-9] which often matches a lot more characters).
  • + matches one or more of the preceding atom.
  • \K resets the start of the matched portion (what to Keep, or in the case of grep -o what to output).

So, that will print all the sequences of one or more digits that follow foo= with any amount of whitespace allowed on either side of the =, even if more than one occur on a given line.

With pcregrep, you can also specify a number after -o to print what's matched by the give capture group, instead of the whole matched portion:

pcregrep -o1 'foo\s*=\s*(\d+)'

Portably, you can actually use the real thing: perl:

perl -lne 'print $1 for m{foo\s*=\s*(\d+)}g'
1
  • Cool, I'm happy to know how this can be done. – Daniel Kaplan Jun 12 at 23:04
3

Assuming you want to pick out a numeric foo value,

echo 'foo = 1700;' | awk '$1=="foo" {print $NF+0}'
1700

By default awk splits on whitespace (not just a single space). NF is the number of fields, in this case 3; $NF is the string value of the 3rd whitespaced field. The +0 converts this string 1700; to the numeric value 1700.

It will work on lines such as foo = 1700; but not lines such as foo=1700;. I wasn't sure from your question whether you were simply concerned about removing extra whitespace, or that your data might have no whitespace and all, with the = and ; being the only boundary points. If you want to ignore any whitespace, regardless of whether there is some or none, you would be better off with sed,

echo 'foo=1700;' | sed -n 's/^foo *= *//p' | sed -e 's/;$//' -e 's/ *$//'
1700
3

Using gawk:

Assuming file input as:

$ cat input
foo = 1700
foo=17
foo           2000
foobarfoo = 200
foo = foo = 14 243
foo =
200 = foo

This gawk command would do:

awk '{if(match($0, /\<foo\s*=\s*[0-9]+/)){ 
a=substr($0,RSTART,RLENGTH);sub(/foo\s*=\s*/, "",a); print a}}' input

Or with this gawk command:

awk '{if(match($0, /\<foo\s*=\s*[0-9]/))
{ l=RLENGTH-1;match($0, /foo\s*=\s*[0-9]+/)
print substr($0,RSTART+l,RLENGTH-l); }}' input

If pattern is simple as in foo = 1700, i.e., a space after foo and equal sign, then the above command can be shortened:

awk '{if(match($0, /\<foo = [0-9]+/)) print substr($0,RSTART+6,RLENGTH-6);}' input

Another method:

awk '/\<foo\s*=\s*[0-9]+/ {print gensub(/(.*)(foo\s*=\s*)([0-9]+)(.*)/, "\\3", "g") }' input

In this command, all captured groups are replaced by third captured group using backreferencing(\\3). gensub() is a gawk builtin function.

1
  • This is a good reference for me because I always forget how to do it, but to me, its verbosity implies it's the wrong tool for the job. Again, still worth knowing and I upvoted your answer – Daniel Kaplan Jun 12 at 23:06
1
awk '{for(i=1;i<=NF;i++){if($i ~ /foo/ && $0 ~ /foo.*=.*[0-9]*/){gsub(";","",$(i+2));print $(i+2)}}}' filename

Tested and worked fine

0
awk '/foo = /{print $NF}' file

... or if you want to save output to var

VAR=$(awk '/foo = /{print $NF}' file)

2
  • I don't see how this answer relates – Daniel Kaplan Jun 12 at 6:32
  • There is a semi colon at the end of the data that you also extract with this. This would also extract from lines saying barfoo = 1700;. – Kusalananda Jun 12 at 7:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.