Is it possible to make sed doing some logic in the replacement part (regex group)

Question

I have few sed command: to extract relevant information

My file sample.log (format is ncsa.log) looks like:

2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:54:21 +0000] "GET /cxf/myservice01/v1/abc?anyparam=anything&anotherone=another HTTP/1.1" 200 3224 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:54:25 +0000] "GET /cxf/myservice02/v1/XYZ?anyparam=anything&anotherone=another HTTP/1.1" 200 3224 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:56:52 +0000] "GET /cxf/myservice01/v1/rsv/USER02?anyparam=anything&anotherone=another HTTP/1.1" 200 6456 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:58:52 +0000] "GET /cxf/myservice01/v2/upr/USER01?anyparam=anything&anotherone=another HTTP/1.1" 200 2424 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:59:11 +0000] "GET /cxf/myservice02/v1/xyz?anyparam=anything&anotherone=another HTTP/1.1" 200 233 "-" "client name"

This set of piped sed's are extracting the url details I need (first sed: \1 = date in YYYY-MM-DD, \2 = service0x, \3 = trigram, \4 = optionnal entity id, \5 = HTTP response code, \6 = http response size)

more sample.log | sed -r 's#^(...._.._..)_.*/cxf/(myservice.*)/v./(.{3})[/]*([a-Z0-9]*)?.*\sHTTP/1.1.\s(.{3})\s([0-9]*)\s.*#\1;\2;\L\3;\E\4;\5;\6#g'  | sed -r 's!(.*;.*;.{3};)[a-Z0-9]+(;.*;.*)!\1retrieve\2!g' | sed -r 's!(.*);;(.*)!\1;list;\2!g' > request-by-operation.txt

The result needed is the following:

2012_04_01;myservice01;abc;list;200;3224
2012_04_01;myservice02;xyz;list;200;3224
2012_04_01;myservice01;rsv;retrieve;200;6456
2012_04_01;myservice01;upr;retrieve;200;2424
2012_04_01;myservice02;xyz;list;200;233

I did not find another way to convert the list and retrieve operation than using two other sed's piped (that does the job).

I heard sed does not supports commands in the replacement part (on a specific group) something like #\1;\2;\L\3;\Eifnull(\4, "list", "retrieve");\5;\6# but I am wondering if I can still do it another way using only one sed command.

Answer 1

sed can't call commands in the replacement part, but it can run several substitutions. Just putting all the substitutions to one sed seems to work in this case:

sed -r 's#^(...._.._..)_.*/cxf/(myservice.*)/v./(.{3})[/]*([a-Z0-9]*)?.*\sHTTP/1.1.\s(.{3})\s([0-9]*)\s.*#\1;\2;\L\3;\E\4;\5;\6#g;
        s!(.*;.*;.{3};)[a-Z0-9]+(;.*;.*)!\1retrieve\2!g;
        s!(.*);;(.*)!\1;list;\2!g'

Answer 2

You can select whanted parts but can remove unwanted instead:

sed '
    s|_[^_]* /[^/]*/|;|
    s|/[^/]*/\(...\)|;\L\1|
    s|?[^"]*" |;list;|
    s|/.*;|;retrieve;|
    s/ /;/
    s/ .*$//'

Answer 3

GNU sed does have an s///e command, but it sends the whole pattern space to the shell for evaluation:

$ echo "echo hello world" | sed 's/world/foo bar | rev/e'
rab oof olleh

So "world" is replaced by "foo bar | rev". Pattern space now is "echo hello foo bar | rev". This is sent to the shell, and the output is placed in pattern space which is then implicitly printed.

Perl's e flag allows you to focus on just the matched part of the string.

See https://www.gnu.org/software/sed/manual/sed.html#The-_0022s_0022-Command

e

This command allows one to pipe input from a shell command into pattern space. If a substitution was made, the command that is found in pattern space is executed and pattern space is replaced with its output. A trailing newline is suppressed; results are undefined if the command to be executed contains a nul character. This is a GNU sed extension.