Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 3 down vote favorite

From time to time I need to dig through huge log files (several GB unpacked) to debug a specific error.

Now, vim is OK for browsing through the file, but when I need to find something in the file it's completely useless. Is there some tool that could index the log and allow me to search the file faster?

Ideally a command line tool.

EDIT: Just to clarify, tools like ack or grep aren't suitable since I need to examine the context of the matches and -C,-A,-B just aren't good enough. Plus ack and grep are still unusably slow.

share|improve this question
Did you try grep? – Bernhard Feb 27 at 11:47
@Bernhard I actually didn't measure grep vs. vim in terms of speed, but grep is really unsuitable for what I need to do with the files. I need more flexibility than the -C -B and -A options offer. – Let_Me_Be Feb 27 at 11:57
There are plenty of logfile handling tools around, have you looked at what your distribution offers in that area? – vonbrand Feb 27 at 12:19
1  
I've heard good things about ack, although I have yet to try it myself. It purports to be "better than grep", according to the domain name. – swisscheese Feb 27 at 12:19
1  
Did you try less? Searching a file with less is by magnitudes faster compared to vim (even with LargeFile plugin). But I assume the file needs to fit into memory. Furthermore I doubt that it will be faster than grep. – Marco Feb 27 at 12:37
show 4 more comments

2 Answers

up vote 3 down vote

There are really good log indexers a bit bigger than command line level. Commercially, splunk is the big one and hard to beat. Graylog2 is a nice open source implementation.

There is a fair bit of overhead to both setting them up and running them but once you have it a part of your process they're great. Querying and reporting logs from multiple sources is the biggest benefit for multi tiered apps.

As for command line, if you can plug your logging into syslog, the systemd journald implementation allows for indexing on a number of fields as items are logged. Not sure if that would be useful in text processing though, more for the metadata.

Not sure about ad hoc commands, they would still have an extract/load/parse lead time.

That being said, less does most of what I need in a single log file. 

&pattern  grep pattern
/pattern  search forward
?pattern  search backward
shift-f   tail -f

the tail mode can take the previous pattern into consideration as well which is great for watching live logs

share|improve this answer
up vote 2 down vote

Searching through a file with less might be an option. It does not require the files to fit into memory. And it's much faster compared to vim (even when the LargeFile plugin is used).

The downsides are: It will probably not beat grep from a performance standpoint. And most of the alternatives choke if the file size approaches the RAM size.

share|improve this answer
less doesn't require the file to fit into memory, it's made to page huge files. It would benefit from a large file cache though, as any non indexing utility would. – mindthemonkey Feb 27 at 13:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.