AWK: how to extract pattern between 2 custom field separators regardless of pattern's position on the line?

Question

Here is a fragment of a line in file:

LN=FINE FOODS & PHARMACEUTICALS NTM, MIC=XAIM, RIC=FF.MI, SG=MA1

I am interested in extracting the value of tag "MIC", i.e. my desired output is:

XAIM

The whole line is quite long:

20200403: #379 IT0005215329 {CU=EUR, GTPID=144115188076657542, II=IT0005215329, IS=18814564, LN=FINE FOODS & PHARMACEUTICALS NTM, MIC=XAIM, RIC=FF.MI, SG=MA1, SN=801670, STY=ORDINARY, TK="0.0002 to 0.1,0.0005 to 0.2,0.001 to 0.5,0.002 to 1,0.005 to 2,0.01 to 5,0.02 to 10,0.05 to 20,0.1 to 50,0.2 to 100,0.5 to 200,1 to 500,2 to 1000,5 to 2000,10 to 5000,20 to 10000,50 to 20000,100 to 50000,200", TS=FF, TY=S, UQ=1}

The position on the line of the tag "MIC" is not always the same.

I read through quite a few tutorials and it seems that all of their solutions involves creating a custom field separators and then extracting a desired pattern by using pattern's position on the line.

For example, I attempted to follow along the example given in this thread, namely I used this code to extract value from "MIC" tag:

awk 'BEGIN {FS="MIC=|,"} {print $2}' input.txt

I got the following output:

GTPID=144115188076657542

If you check the whole line sample that I provided above, the output is the value of the second tag "GTPID" that has "=" symbol. At first I was thinking that {FS="MIC=|,"} meant "create two custom field separators, the 1st one being MIC= and the 2nd one being , and for some reason I expected that {print $2} will print out whatever is between those two field separators.

But obviously the code above prints the value of whatever pattern that contains symbol "=" happens to be 2nd on the line.

How do I extract value that is between MIC= and , then?

Answer 1

Whenever you have name=value pairs in your data it's best to first create an array that captures that mapping (f[] below) and then you can just access whatever fields you like by their name(s), e.g.:

$ awk -F'[=,] *' '{for (i=1;i<NF;i+=2) f[$i]=$(i+1); print f["MIC"]}' file
XAIM

Look at how easy that is to adapt to test values, print other fields in any order, etc.:

awk -F'[=,] *' '
    { for (i=1;i<NF;i+=2) f[$i]=$(i+1) }
    (f["MIC"] == "XAIM") && (f["LN"] ~ /FOOD/){ print f["SG"], f["RIC"] }
' file
MA1 FF.MI

Answer 2

$ sed -n 's/.* MIC=\([^,}]*\).*/\1/p' file
XAIM

This uses sed to match the  MIC=SOMETHING, or MIC=SOMETHING} string, and replaces the whole line with the SOMETHING string. All other data is discarded.

---

$ tr ',' '\n' <file | awk -F '=' '$1 == " MIC" { print $2 }'
XAIM

This first replaces all commas by newlines and then runs awk with a = character as field delimiter. When the first field is equal to  MIC, the second field is printed.

---

$ awk -F ',' '{ for (i = 1; i <= NF; ++i) if (sub(" MIC=","",$i)) print $i }' file
XAIM

This only uses awk and treats the input as comma separated fields. It iterates over all fields, and when a field starts with the string  MIC=, that string is removed from the field and the remainder is printed.

---

If the file had been in JSON format (I'm thinking you may have transformed the data from JSON at some point as most REST APIs return JSON formatted DATA, and this data seems to be related to financial stock markets):

{
  "CU": "EUR",
  "GTPID": 144115188076657540,
  "II": "IT0005215329",
  "IS": 18814564,
  "LN": "FINE FOODS & PHARMACEUTICALS NTM",
  "MIC": "XAIM",
  "RIC": "FF.MI",
  "SG": "MA1",
  "SN": 801670,
  "STY": "ORDINARY",
  "TK": "0.0002 to 0.1,0.0005 to 0.2,0.001 to 0.5,0.002 to 1,0.005 to 2,0.01 to 5,0.02 to 10,0.05 to 20,0.1 to 50,0.2 to 100,0.5 to 200,1 to 500,2 to 1000,5 to 2000,10 to 5000,20 to 10000,50 to 20000,100 to 50000,200",
  "TS": "FF",
  "TY": "S",
  "UQ": 1
}

then jq would have been easiest:

$ jq -r '.MIC' file1
XAIM

Answer 3

command

 awk -F "," '{for(i=1;i<=NF;i++){if($i ~ /MIC/){gsub(/.*=/,"",$i);print $i}}}' 

filename

output

XAIM

Answer 4

With grep and cut. Use grep -o to take only the matched data, look for the requested field and value. Feed that to cut, using = as a field seperator, and take the second field:

$ grep -o 'MIC=[^,]*' input | cut -d= -f2
XAIM

With sed. Look for the requested field/value pair, use () and \1 to extract the matching subpattern:

$ sed -nE 's/^.*MIC=([^,]+).*$/\1/;p' input
XAIM
# or, alternatively,
$ sed -n 's/^.*MIC=\([^,]*\).*$/\1/;p' input
XAIM

With awk. Set the field separator and record separator to = and , respectively. For the record with the matching pattern, print the second field (i. e. the value):

$ awk 'BEGIN { FS="="; RS=","; } $1 ~ /MIC/ { print $2 }' input
XAIM