I have few sed command: to extract relevant information

My file sample.log (format is ncsa.log) looks like:

2012_04_01_filename.log:29874: - - [16/Aug/2012:12:54:21 +0000] "GET /cxf/myservice01/v1/abc?anyparam=anything&anotherone=another HTTP/1.1" 200 3224 "-" "client name"
2012_04_01_filename.log:29874: - - [16/Aug/2012:12:54:25 +0000] "GET /cxf/myservice02/v1/XYZ?anyparam=anything&anotherone=another HTTP/1.1" 200 3224 "-" "client name"
2012_04_01_filename.log:29874: - - [16/Aug/2012:12:56:52 +0000] "GET /cxf/myservice01/v1/rsv/USER02?anyparam=anything&anotherone=another HTTP/1.1" 200 6456 "-" "client name"
2012_04_01_filename.log:29874: - - [16/Aug/2012:12:58:52 +0000] "GET /cxf/myservice01/v2/upr/USER01?anyparam=anything&anotherone=another HTTP/1.1" 200 2424 "-" "client name"
2012_04_01_filename.log:29874: - - [16/Aug/2012:12:59:11 +0000] "GET /cxf/myservice02/v1/xyz?anyparam=anything&anotherone=another HTTP/1.1" 200 233 "-" "client name"

This set of piped sed's are extracting the url details I need (first sed: \1 = date in YYYY-MM-DD, \2 = service0x, \3 = trigram, \4 = optionnal entity id, \5 = HTTP response code, \6 = http response size)

more sample.log | sed -r 's#^(...._.._..)_.*/cxf/(myservice.*)/v./(.{3})[/]*([a-Z0-9]*)?.*\sHTTP/1.1.\s(.{3})\s([0-9]*)\s.*#\1;\2;\L\3;\E\4;\5;\6#g'  | sed -r 's!(.*;.*;.{3};)[a-Z0-9]+(;.*;.*)!\1retrieve\2!g' | sed -r 's!(.*);;(.*)!\1;list;\2!g' > request-by-operation.txt

The result needed is the following:


I did not find another way to convert the list and retrieve operation than using two other sed's piped (that does the job).

I heard sed does not supports commands in the replacement part (on a specific group) something like #\1;\2;\L\3;\Eifnull(\4, "list", "retrieve");\5;\6# but I am wondering if I can still do it another way using only one sed command.

  • Do you have to use sed? Sounds like a job for perl.
    – muru
    Commented Nov 19, 2015 at 8:08
  • Are you examining old logs or are you actively running a web server that has seen no active development in over a decade? If these logs are still being generated you might want to look into the options to change the log format to match what you want or at least match what you want closer.
    – Bram
    Commented Nov 19, 2015 at 8:28
  • @muru no I am not strict to use only sed but my question was mainly if I can "squash" a bit the command line with all the pipes.
    – рüффп
    Commented Nov 19, 2015 at 9:26
  • @Bram these are the ncsa-log automatically generated by ServiceMix. We have another log files with a specific format. We did not check if we can modify this format by configuration (jetty, pax.web, no idea where) and this is the default format and yes ServiceMix seems to be still active in the community :)
    – рüффп
    Commented Nov 19, 2015 at 9:30

3 Answers 3


sed can't call commands in the replacement part, but it can run several substitutions. Just putting all the substitutions to one sed seems to work in this case:

sed -r 's#^(...._.._..)_.*/cxf/(myservice.*)/v./(.{3})[/]*([a-Z0-9]*)?.*\sHTTP/1.1.\s(.{3})\s([0-9]*)\s.*#\1;\2;\L\3;\E\4;\5;\6#g;
  • Still not so compact but good to know I can put all replacement in one sed only.
    – рüффп
    Commented Nov 19, 2015 at 13:34

You can select whanted parts but can remove unwanted instead:

sed '
    s|_[^_]* /[^/]*/|;|
    s|?[^"]*" |;list;|
    s/ /;/
    s/ .*$//'
  • Is there any reason why you use | as replacement separator first and then /? except some patterns contains a slash (but not the third one).
    – рüффп
    Commented Nov 20, 2015 at 8:53

GNU sed does have an s///e command, but it sends the whole pattern space to the shell for evaluation:

$ echo "echo hello world" | sed 's/world/foo bar | rev/e'
rab oof olleh

So "world" is replaced by "foo bar | rev". Pattern space now is "echo hello foo bar | rev". This is sent to the shell, and the output is placed in pattern space which is then implicitly printed.

Perl's e flag allows you to focus on just the matched part of the string.

See https://www.gnu.org/software/sed/manual/sed.html#The-_0022s_0022-Command


This command allows one to pipe input from a shell command into pattern space. If a substitution was made, the command that is found in pattern space is executed and pattern space is replaced with its output. A trailing newline is suppressed; results are undefined if the command to be executed contains a nul character. This is a GNU sed extension.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .