1

I have few sed command: to extract relevant information

My file sample.log (format is ncsa.log) looks like:

2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:54:21 +0000] "GET /cxf/myservice01/v1/abc?anyparam=anything&anotherone=another HTTP/1.1" 200 3224 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:54:25 +0000] "GET /cxf/myservice02/v1/XYZ?anyparam=anything&anotherone=another HTTP/1.1" 200 3224 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:56:52 +0000] "GET /cxf/myservice01/v1/rsv/USER02?anyparam=anything&anotherone=another HTTP/1.1" 200 6456 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:58:52 +0000] "GET /cxf/myservice01/v2/upr/USER01?anyparam=anything&anotherone=another HTTP/1.1" 200 2424 "-" "client name"
2012_04_01_filename.log:29874:192.168.1.12 - - [16/Aug/2012:12:59:11 +0000] "GET /cxf/myservice02/v1/xyz?anyparam=anything&anotherone=another HTTP/1.1" 200 233 "-" "client name"

This set of piped sed's are extracting the url details I need (first sed: \1 = date in YYYY-MM-DD, \2 = service0x, \3 = trigram, \4 = optionnal entity id, \5 = HTTP response code, \6 = http response size)

more sample.log | sed -r 's#^(...._.._..)_.*/cxf/(myservice.*)/v./(.{3})[/]*([a-Z0-9]*)?.*\sHTTP/1.1.\s(.{3})\s([0-9]*)\s.*#\1;\2;\L\3;\E\4;\5;\6#g'  | sed -r 's!(.*;.*;.{3};)[a-Z0-9]+(;.*;.*)!\1retrieve\2!g' | sed -r 's!(.*);;(.*)!\1;list;\2!g' > request-by-operation.txt

The result needed is the following:

2012_04_01;myservice01;abc;list;200;3224
2012_04_01;myservice02;xyz;list;200;3224
2012_04_01;myservice01;rsv;retrieve;200;6456
2012_04_01;myservice01;upr;retrieve;200;2424
2012_04_01;myservice02;xyz;list;200;233

I did not find another way to convert the list and retrieve operation than using two other sed's piped (that does the job).

I heard sed does not supports commands in the replacement part (on a specific group) something like #\1;\2;\L\3;\Eifnull(\4, "list", "retrieve");\5;\6# but I am wondering if I can still do it another way using only one sed command.

  • Do you have to use sed? Sounds like a job for perl. – muru Nov 19 '15 at 8:08
  • Are you examining old logs or are you actively running a web server that has seen no active development in over a decade? If these logs are still being generated you might want to look into the options to change the log format to match what you want or at least match what you want closer. – Bram Nov 19 '15 at 8:28
  • @muru no I am not strict to use only sed but my question was mainly if I can "squash" a bit the command line with all the pipes. – рüффп Nov 19 '15 at 9:26
  • @Bram these are the ncsa-log automatically generated by ServiceMix. We have another log files with a specific format. We did not check if we can modify this format by configuration (jetty, pax.web, no idea where) and this is the default format and yes ServiceMix seems to be still active in the community :) – рüффп Nov 19 '15 at 9:30
1

sed can't call commands in the replacement part, but it can run several substitutions. Just putting all the substitutions to one sed seems to work in this case:

sed -r 's#^(...._.._..)_.*/cxf/(myservice.*)/v./(.{3})[/]*([a-Z0-9]*)?.*\sHTTP/1.1.\s(.{3})\s([0-9]*)\s.*#\1;\2;\L\3;\E\4;\5;\6#g;
        s!(.*;.*;.{3};)[a-Z0-9]+(;.*;.*)!\1retrieve\2!g;
        s!(.*);;(.*)!\1;list;\2!g'
  • Still not so compact but good to know I can put all replacement in one sed only. – рüффп Nov 19 '15 at 13:34
1

You can select whanted parts but can remove unwanted instead:

sed '
    s|_[^_]* /[^/]*/|;|
    s|/[^/]*/\(...\)|;\L\1|
    s|?[^"]*" |;list;|
    s|/.*;|;retrieve;|
    s/ /;/
    s/ .*$//'
  • Is there any reason why you use | as replacement separator first and then /? except some patterns contains a slash (but not the third one). – рüффп Nov 20 '15 at 8:53
0

GNU sed does have an s///e command, but it sends the whole pattern space to the shell for evaluation:

$ echo "echo hello world" | sed 's/world/foo bar | rev/e'
rab oof olleh

So "world" is replaced by "foo bar | rev". Pattern space now is "echo hello foo bar | rev". This is sent to the shell, and the output is placed in pattern space which is then implicitly printed.

Perl's e flag allows you to focus on just the matched part of the string.

See https://www.gnu.org/software/sed/manual/sed.html#The-_0022s_0022-Command

e

This command allows one to pipe input from a shell command into pattern space. If a substitution was made, the command that is found in pattern space is executed and pattern space is replaced with its output. A trailing newline is suppressed; results are undefined if the command to be executed contains a nul character. This is a GNU sed extension.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.