0

Here is a fragment of a line in file:

LN=FINE FOODS & PHARMACEUTICALS NTM, MIC=XAIM, RIC=FF.MI, SG=MA1

I am interested in extracting the value of tag "MIC", i.e. my desired output is:

XAIM

The whole line is quite long:

20200403: #379 IT0005215329 {CU=EUR, GTPID=144115188076657542, II=IT0005215329, IS=18814564, LN=FINE FOODS & PHARMACEUTICALS NTM, MIC=XAIM, RIC=FF.MI, SG=MA1, SN=801670, STY=ORDINARY, TK="0.0002 to 0.1,0.0005 to 0.2,0.001 to 0.5,0.002 to 1,0.005 to 2,0.01 to 5,0.02 to 10,0.05 to 20,0.1 to 50,0.2 to 100,0.5 to 200,1 to 500,2 to 1000,5 to 2000,10 to 5000,20 to 10000,50 to 20000,100 to 50000,200", TS=FF, TY=S, UQ=1}

The position on the line of the tag "MIC" is not always the same.

I read through quite a few tutorials and it seems that all of their solutions involves creating a custom field separators and then extracting a desired pattern by using pattern's position on the line.

For example, I attempted to follow along the example given in this thread, namely I used this code to extract value from "MIC" tag:

awk 'BEGIN {FS="MIC=|,"} {print $2}' input.txt

I got the following output:

GTPID=144115188076657542

If you check the whole line sample that I provided above, the output is the value of the second tag "GTPID" that has "=" symbol. At first I was thinking that {FS="MIC=|,"} meant "create two custom field separators, the 1st one being MIC= and the 2nd one being , and for some reason I expected that {print $2} will print out whatever is between those two field separators.

But obviously the code above prints the value of whatever pattern that contains symbol "=" happens to be 2nd on the line.

How do I extract value that is between MIC= and , then?

1
  • The right answer would depend on whether your input is more complicated than you, and whether your script is more complicated than you show. For what you have there, sed -n 's/.*MIC=\([^,]*\).*/\1/p' would do. Apr 6, 2020 at 20:11

4 Answers 4

1

Whenever you have name=value pairs in your data it's best to first create an array that captures that mapping (f[] below) and then you can just access whatever fields you like by their name(s), e.g.:

$ awk -F'[=,] *' '{for (i=1;i<NF;i+=2) f[$i]=$(i+1); print f["MIC"]}' file
XAIM

Look at how easy that is to adapt to test values, print other fields in any order, etc.:

awk -F'[=,] *' '
    { for (i=1;i<NF;i+=2) f[$i]=$(i+1) }
    (f["MIC"] == "XAIM") && (f["LN"] ~ /FOOD/){ print f["SG"], f["RIC"] }
' file
MA1 FF.MI
4
  • Except that won't print anything at all for f["CU"] and will print bogus values for f["UQ"] or f["TK"] in the OP's example. It also won't "adapt" at all if the TK="..." happens to contain an odd number of commas -- your example generates bogus array entries like f["0.0005 to 0.2"]="0.001 to 0.5". While awk is powerful enough to parse such recursive/structured data, pseudo-solutions like this give it a bad rap. Apr 8, 2020 at 20:26
  • It's a solution for the problem the OP has, not an all inclusive solution for every possible variation of the problem that the OP doesn't have. Why not start ranting about it not handling newlines or escaped quotes inside quoted strings while you're at it? Why not assume there might be some variation that uses {} or some other structure to create hierarchies of fields and then complain it doesn't handle that either?
    – Ed Morton
    Apr 8, 2020 at 21:11
  • Behave, don't call people "ranting" for pointing out how broken and ridiculous your stuff is. Yes, why not handling escape quotes inside quoted strings? Your solution is no better than awk '{print"XAIM"}'. Apr 8, 2020 at 21:19
  • awk '{print"XAIM"}' would output XAIM even if MIC wasn't present in the input or was assigned a different value like MIC=RANT. There are all sorts of improvements we COULD make for efficiency, robustness, etc. but obviously since they accepted it the script I posted is adequate for the OPs needs so, as is frequently the case, there's no need to go writing a lengthier script that solves problems the OP simply doesn't have.
    – Ed Morton
    Apr 9, 2020 at 15:24
1
$ sed -n 's/.* MIC=\([^,}]*\).*/\1/p' file
XAIM

This uses sed to match the  MIC=SOMETHING, or MIC=SOMETHING} string, and replaces the whole line with the SOMETHING string. All other data is discarded.


$ tr ',' '\n' <file | awk -F '=' '$1 == " MIC" { print $2 }'
XAIM

This first replaces all commas by newlines and then runs awk with a = character as field delimiter. When the first field is equal to  MIC, the second field is printed.


$ awk -F ',' '{ for (i = 1; i <= NF; ++i) if (sub(" MIC=","",$i)) print $i }' file
XAIM

This only uses awk and treats the input as comma separated fields. It iterates over all fields, and when a field starts with the string  MIC=, that string is removed from the field and the remainder is printed.


If the file had been in JSON format (I'm thinking you may have transformed the data from JSON at some point as most REST APIs return JSON formatted DATA, and this data seems to be related to financial stock markets):

{
  "CU": "EUR",
  "GTPID": 144115188076657540,
  "II": "IT0005215329",
  "IS": 18814564,
  "LN": "FINE FOODS & PHARMACEUTICALS NTM",
  "MIC": "XAIM",
  "RIC": "FF.MI",
  "SG": "MA1",
  "SN": 801670,
  "STY": "ORDINARY",
  "TK": "0.0002 to 0.1,0.0005 to 0.2,0.001 to 0.5,0.002 to 1,0.005 to 2,0.01 to 5,0.02 to 10,0.05 to 20,0.1 to 50,0.2 to 100,0.5 to 200,1 to 500,2 to 1000,5 to 2000,10 to 5000,20 to 10000,50 to 20000,100 to 50000,200",
  "TS": "FF",
  "TY": "S",
  "UQ": 1
}

then jq would have been easiest:

$ jq -r '.MIC' file1
XAIM
1

command

 awk -F "," '{for(i=1;i<=NF;i++){if($i ~ /MIC/){gsub(/.*=/,"",$i);print $i}}}' 

filename

output

XAIM
0

With grep and cut. Use grep -o to take only the matched data, look for the requested field and value. Feed that to cut, using = as a field seperator, and take the second field:

$ grep -o 'MIC=[^,]*' input | cut -d= -f2
XAIM

With sed. Look for the requested field/value pair, use () and \1 to extract the matching subpattern:

$ sed -nE 's/^.*MIC=([^,]+).*$/\1/;p' input
XAIM
# or, alternatively,
$ sed -n 's/^.*MIC=\([^,]*\).*$/\1/;p' input
XAIM

With awk. Set the field separator and record separator to = and , respectively. For the record with the matching pattern, print the second field (i. e. the value):

$ awk 'BEGIN { FS="="; RS=","; } $1 ~ /MIC/ { print $2 }' input
XAIM
13
  • grep -Po '\bMIC=\K[^,]*' Apr 6, 2020 at 20:16
  • 1
    I usually take careful heed of warnings in manual pages, in relevant case that --perl-regexp is "highly experimental".
    – DopeGhoti
    Apr 6, 2020 at 20:18
  • 1
    And yet the admonition in the manual page persists.
    – DopeGhoti
    Apr 6, 2020 at 20:22
  • 1
    Interesting suggestion. Usually the advice is to RTFM, not to ITFM.
    – DopeGhoti
    Apr 6, 2020 at 20:25
  • 1
    @kamokoba perl's regexes are able to parse such recursive data formats. However, they're not for the faint of heart ;-) I'll give you a example when I get a device with an actual keyboard. Apr 21, 2020 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.