How to grep between words in a log?

Question

I have a txt file with lines like below.

Starting With:

 Parameters: {"raw_message"=>"MSH.......

Ending With:

</HL7Message>"}

I'm wanting to grep and output to a file the words between raw and transformed which appears halfway through each line.

The file appears like below

Parameters: {"raw_message"=>"MSH....... "transformed_data".....</HL7Message>"}

Parameters: {"raw_message"=>"MSH....... "transformed_data".....</HL7Message>"}

Parameters: {"raw_message"=>"MSH....... "transformed_data".....</HL7Message>"}

Parameters: {"raw_message"=>"MSH....... "transformed_data".....</HL7Message>"}

Best case scenario

The MSH following the >MSH begins the output but there are many MSH instances in each line. So i figured it could be logical to grep the between message and then rip the raw and transformed part out.

raw_message"=>"MSH......preceding words followed by transformed

Some possible words preceding transformed LAB\r", "transformed 00355", "transformed

So I'd want

MSH....LAB\r
MSH....00355

Any assistance would be much appreciated!

I tried:

sed -n "/<raw>/,/<\/transformed>/p" HL7prod.txt > HL7prod2.txt

Example Line

 Parameters: {"raw_message"=>"MSH|^~\\&||CDFGTL|||20144543000||ATG^A05|TLGTADM.1.13773085|P|2.1\rEVN|A08|11111111111|||MDFGQ8833^HLPS^GEGES^^^^\rPID|1||K11111111|K1111111|HOLVBVFS^LGDSA^^^^||19GHYSSD|F|^^^^^||^^^^^^^^|||||||K01045435547691\rPV1|1|P|K.ER^^||||LKIJK^Liujn^Jeggrs^H^^^MD|||ER||||||N||ER|||||||||||||||||||||DFLHL|ABD DFIN|PRE|||111111111||||||||\rZCS||^^^^||||00355", "transformed_data"=>"<HL7Message><MSH><MSH.1>|</MSH.1><MSH.2>^~\\&amp;</MSH.2><MSH.3><MSH.3.1>CDFLH</MSH.3.1></MSH.3><MSH.4><MSH.4.1>COCTL</MSH.4.1></MSH.4><MSH.5/><MSH.6/><MSH.7><MSH.7.1>201506331000</MSH.7.1></MSH.7><MSH.8/><MSH.9><MSH.9.1>ADT</MSH.9.1><MSH.9.2>A08</MSH.9.2></MSH.9><MSH.10><MSH.10.1>TLGGBGM.1.13773076</MSH.10.1></MSH.10><MSH.11><MSH.11.1>P</MSH.11.1></MSH.11><MSH.12><MSH.12.1>2.1</MSH.12.1></MSH.12></MSH><EVN><EVN.1><EVN.1.1>A08</EVN.1.1></EVN.1><EVN.2><EVN.2.1>201506125500</EVN.2.1></EVN.2><EVN.3/><EVN.4/><EVN.5><EVN.5.1>MDHYQ6633</EVN.5.1><EVN.5.2>LUJKL</EVN.5.2><EVN.5.3>JYTEDFG</EVN.5.3><EVN.5.4/><EVN.5.5/><EVN.5.6/><EVN.5.7/></EVN.5></EVN><PID><PID.1><PID.1.1>1</PID.1.1></PID.1><PID.2/><PID.3><PID.3.1>K0567432372</PID.3.1></PID.3><PID.4><PID.4.1>K5894336</PID.4.1></PID.4><PID.5><PID.5.1>HOLDFGEER</PID.5.1><PID.5.2>AAAAS</PID.5.2><PID.5.3/><PID.5.4/><PID.5.5/><PID.5.6/></PID.5><PID.6/><PID.7><PID.7.1>1111111111</PID.7.1></PID.7><PID.8><PID.8.1>F</PID.8.1></PID.8><PID.9><PID.9.1/><PID.9.2/><PID.9.3/><PID.9.4/><PID.9.5/><PID.9.6/></PID.9><PID.10/><PID.11><PID.11.1/><PID.11.2/><PID.11.3/><PID.11.4/><PID.11.5/><PID.11.6/><PID.11.7/><PID.11.8/><PID.11.9/></PID.11><PID.12/><PID.13><PID.13.1/></PID.13><PID.14/><PID.15/><PID.16/><PID.17/><PID.18><PID.18.1>K0101333333333</PID.18.1></PID.18></PID><PV1><PV1.1><PV1.1.1>1</PV1.1.1></PV1.1><PV1.2><PV1.2.1>P</PV1.2.1></PV1.2><PV1.3><PV1.3.1>K.ER</PV1.3.1><PV1.3.2/><PV1.3.3/></PV1.3><PV1.4/><PV1.5/><PV1.6/><PV1.7><PV1.7.1>JTOLOKS</PV1.7.1><PV1.7.2>Ldasfs</PV1.7.2><PV1.7.3>Jtuygikd</PV1.7.3><PV1.7.4>H</PV1.7.4><PV1.7.5/><PV1.7.6/><PV1.7.7>MD</PV1.7.7></PV1.7><PV1.8/><PV1.9/><PV1.10><PV1.10.1>ER</PV1.10.1></PV1.10><PV1.11/><PV1.12/><PV1.13/><PV1.14/><PV1.15/><PV1.16><PV1.16.1>N</PV1.16.1></PV1.16><PV1.17/><PV1.18><PV1.18.1>ER</PV1.18.1></PV1.18><PV1.19/><PV1.20/><PV1.21/><PV1.22/><PV1.23/><PV1.24/><PV1.25/><PV1.26/><PV1.27/><PV1.28/><PV1.29/><PV1.30/><PV1.31/><PV1.32/><PV1.33/><PV1.34/><PV1.35/><PV1.36/><PV1.37/><PV1.38/><PV1.39><PV1.39.1>COTOLA</PV1.39.1></PV1.39><PV1.40><PV1.40.1>ABD XXXX</PV1.40.1></PV1.40><PV1.41><PV1.41.1>PRE</PV1.41.1></PV1.41><PV1.42/><PV1.43/><PV1.44><PV1.44.1>111111111</PV1.44.1></PV1.44><PV1.45/><PV1.46/><PV1.47/><PV1.48/><PV1.49/><PV1.50/><PV1.51/><PV1.52/></PV1><ZCS><ZCS.1/><ZCS.2><ZCS.2.1/><ZCS.2.2/><ZCS.2.3/><ZCS.2.4/><ZCS.2.5/></ZCS.2><ZCS.3/><ZCS.4/><ZCS.5/><ZCS.6><ZCS.6.1>111111</ZCS.6.1></ZCS.6></ZCS><GT1><GT1.6><GT1.6.1/></GT1.6></GT1><ZRF><ZRF.1><ZRF.1.1>COTYUL</ZRF.1.1></ZRF.1><ZRF.2><ZRF.2.1>CDFTL</ZRF.2.1><ZRF.2.2>K.ER</ZRF.2.2></ZRF.2></ZRF></HL7Message>"}

Would Want:

MSH|^~\\&||CDFGTL|||20144543000||ATG^A05|TLGTADM.1.13773085|P|2.1\rEVN|A08|11111111111|||MDFGQ8833^HLPS^GEGES^^^^\rPID|1||K11111111|K1111111|HOLVBVFS^LGDSA^^^^||19GHYSSD|F|^^^^^||^^^^^^^^|||||||K01045435547691\rPV1|1|P|K.ER^^||||LKIJK^Liujn^Jeggrs^H^^^MD|||ER||||||N||ER|||||||||||||||||||||DFLHL|ABD DFIN|PRE|||25679506645657||||||||\rZCS||^^^^||||00355

Answer 1

If you want just the text between your patterns on each line, do the following:

sed 's/.*raw\(.*\)transformed.*/\1/'

\(.*\) remembers the text that is output using \1. Other stuff onthe line is not output.

Answer 2

If you have a grep that supports PCREs, you can do

$ grep -oP 'Parameters: {"raw_message"=>"\K.+?(?=", "transformed_data")' file

Or, if your files are as you show, simplify to:

$ grep -oP '=>"\K.+?(?=",)' file

Alternatively, use awk, set the field separator to an => or a , and print the second field (this also matches the ", pipe through tr -d"` to remove them):

$ awk -F'[,>]' '{print $2}' file

Or, you could just grep for the first quoted string starting with MSH:

$ grep -Po '"MSH.+?"' file

And, avoiding the quotes:

$ grep -Po '"\KMSH.*?(?=")' file

Answer 3

The quotes in your Example Line do seem to be fairly significant. If there is no chance of an escaped quote in that string, then all you have to do is:

cut -d\" -f4 <in >out

To get the fourth field on a line as separated by "double-quote delimiters.

If double-quotes might occur escaped in that string - such as by backslashes - then you could do:

sed 's/[^>]*."//;s/", ".*//' 

...to get safely get only the field you want without skipping anything and without false positives. The above two substitutions should handle that pretty robustly if those are the only quote characters (not counting backslashes) which might be used, and given all of the raw bits start like > and that there is no > occurring anywhere before it.