Imagine the following problem: I have a standard Apache log. I want to filter out request URLs with a certain keyword. Simple, we use,
$ grep "keyword" somedomain-access.log
This works well, but isn't robust, i.e., it doesn't work as required in 100% of the cases. Imagine our keyword being "GET", (depending on the website) almost all lines would be matched by the grep command.
We do some preprocessing by doing:
$ cut -d" " -f['i'th field containing the url] somedomain-access.log | grep "keyword"
This works a bit better, but now I lose all context information, i.e., the other fields which I threw away with the
Now this issue in general is biting me more and more. Of course you can use awk in this specific case, but often in the general case an
awk command doesn't suffice.
In non-shell languages you'd simply check for a object's member and do your operation. So using our Apache log example again, take this in Scala,
> // Preprocessed list of apache log into `records` > val records = List(AccessLogRecord(...), AccessLogRecord(...), ...) > val recordsWithKeyword = records.filter(_ => _.request == "keyword")
The result is still a list where every item, in this case AccessLogRecord, still has all context information like date, HTTP code, etc.
How can we achieve the same on the command line using pipes, redirection, etc; without relying on specific applications like