Imagine the following problem: I have a standard Apache log. I want to filter out request URLs with a certain keyword. Simple, we use,

$ grep "keyword" somedomain-access.log

This works well, but isn't robust, i.e., it doesn't work as required in 100% of the cases. Imagine our keyword being "GET", (depending on the website) almost all lines would be matched by the grep command.

We do some preprocessing by doing:

$ cut -d" " -f['i'th field containing the url] somedomain-access.log | 
    grep "keyword"

This works a bit better, but now I lose all context information, i.e., the other fields which I threw away with the cut command.

Now this issue in general is biting me more and more. Of course you can use awk in this specific case, but often in the general case an awk command doesn't suffice.

In non-shell languages you'd simply check for a object's member and do your operation. So using our Apache log example again, take this in Scala,

> // Preprocessed list of apache log into `records`
> val records = List(AccessLogRecord(...), AccessLogRecord(...), ...)
> val recordsWithKeyword = records.filter(_ => _.request == "keyword")

The result is still a list where every item, in this case AccessLogRecord, still has all context information like date, HTTP code, etc.

How can we achieve the same on the command line using pipes, redirection, etc; without relying on specific applications like awk?

  • 4
    Often in the general case a proper awk command does suffice. If you don't want to use the right tools for this job you can always try to do it the hard way that is figure out a regex that matches "keyword" only on the i th field... – don_crissti Jan 15 '17 at 16:34
  • indeed, if your apache logs aren't formatted sufficiently to differentiate fields, then you could modify the log format accordingly – Jeff Schaller Jan 15 '17 at 17:16
  • My requirement is not the usage of grep per se. I'm just seeing a recurring problem in long pipelines on the command line, whereas I do not see such problems in programming languages. I see I've done a bad job at describing the deeper problem. I'll update the question as soon as I can layer my finger on the exact problem, without resorting to examples which apparently distract from the bigger picture. – hbogert Jan 15 '17 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.