4

From time to time I need to dig through huge log files (several GB unpacked) to debug a specific error.

Now, vim is OK for browsing through the file, but when I need to find something in the file it's completely useless. Is there some tool that could index the log and allow me to search the file faster?

Ideally a command line tool.

EDIT: Just to clarify, tools like ack or grep aren't suitable since I need to examine the context of the matches and -C,-A,-B just aren't good enough. Plus ack and grep are still unusably slow.

9
  • Did you try grep?
    – Bernhard
    Feb 27 '13 at 11:47
  • @Bernhard I actually didn't measure grep vs. vim in terms of speed, but grep is really unsuitable for what I need to do with the files. I need more flexibility than the -C -B and -A options offer. Feb 27 '13 at 11:57
  • There are plenty of logfile handling tools around, have you looked at what your distribution offers in that area?
    – vonbrand
    Feb 27 '13 at 12:19
  • 1
    I've heard good things about ack, although I have yet to try it myself. It purports to be "better than grep", according to the domain name. Feb 27 '13 at 12:19
  • 1
    Did you try less? Searching a file with less is by magnitudes faster compared to vim (even with LargeFile plugin). But I assume the file needs to fit into memory. Furthermore I doubt that it will be faster than grep.
    – Marco
    Feb 27 '13 at 12:37
4

There are really good log indexers that are a bit bigger than command line tool level. Commercially, splunk is the big one and hard to beat. Graylog2 is a nice open source implementation. Elasticsearch+Logstash are quite good too.

There is a fair bit of overhead to both setting them up to collect and running the collection server but once you have central log collection as a part of your system, they're great. Querying and reporting logs from multiple sources is the biggest benefit for multi tiered apps.

As for command line, if you can plug your logging into syslog, the systemd journald implementation allows for indexing on a number of fields as items are logged. Not sure if that would be useful in text processing though, more for the metadata.

Not sure about any ad hoc command line tools, they would still have an extract/load/parse time penalty each time you load the file.

That being said, less does most of what I need in a single log file.

&pattern  grep pattern
/pattern  search forward
?pattern  search backward
shift-f   tail -f

the tail mode can take the previous pattern into consideration as well which is great for watching live logs

2

Searching through a file with less might be an option. It does not require the files to fit into memory. And it's much faster compared to vim (even when the LargeFile plugin is used).

The downsides are: It will probably not beat grep from a performance standpoint. And most of the alternatives choke if the file size approaches the RAM size.

1
  • less doesn't require the file to fit into memory, it's made to page huge files. It would benefit from a large file cache though, as any non indexing utility would.
    – Matt
    Feb 27 '13 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.