Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there an easy way to truncate a big log file? Or in other words: How to view only special contents of a log file. E.g.

  • How can I extract the last 1000 lines from a log file?
  • How can I extract only lines with a special string in it?
  • How can I extract log files with a special date/time?

I only know tail but this command is restricted to the last 10 lines and it is too fast. Searching in a 8 GB log file takes to much time ...

share|improve this question
add comment

4 Answers 4

up vote 3 down vote accepted

How can I extract the last 1000 lines from a log file?

tail -1000 file.log

How can I extract only lines with a special string in it?

grep special_string file.log

How can I extract log files with a special date/time?

I'm not sure I understood this one, can you elaborate more? What do you need to do exactly?

share|improve this answer
    
Yeah, I tried the grep command, but with a 8 GB log file this doesn't work. At your last point: I want for example the log entries from 12th October 2pm to 12th October 3pm. –  testing Oct 12 '11 at 17:44
    
grep will work on an 8 GB log file. You may need to wait. You may need to take care how you specify your special string. –  RedGrittyBrick Oct 14 '11 at 10:53
    
@testing: I just saw this comment. About the third thing you asked, you can definitely do it with the find command. The syntax depends on the machine you're on, since different Unix have different implementations. Your best bet: man find! –  rahmu Jan 20 '12 at 12:09
    
@testing: I'm assuming you mean you're looking for the log files that have been last modified in special times. –  rahmu Jan 20 '12 at 12:11
    
@rahmu: Yes, I look for entries in the log files and I want to constrain the set of result depending on a certain period of time. I'll look at the find command the next time. –  testing Jan 20 '12 at 13:52
add comment

How can I extract only lines with a special string in it?

I'll only address this one. On Solaris or HP-UX systems grep, egrep and fgrep exist as seperate files, and each using a different algorithm. Most linux distros just have GNU grep, and symbolic links to each of the 3 names. If you don't have to match a pattern with a metacharacter ('.', '*', '?', etc) in it, just match a plain string, then fgrep will undoubtedly run faster. Even if fgrep runs faster for you, 8GB of log is an awful lot of text to process. How about splitting it up into smaller, half-gig files?

share|improve this answer
    
Is there a convenient command for splitting a 8 GB file in half-gig files? –  testing Oct 26 '11 at 16:25
1  
Using GNU split: split -b 500m EightGBfile.txt –  Bruce Ediger Oct 26 '11 at 16:41
add comment

If you want to do it once, use grep and tail.

Consider using logrotate (from a cron to deal with huge log files. You can configure it to split logs by time, size, number of lines etc. It's very handy and flexible.

share|improve this answer
add comment

Also consider logwatch or something similar to filter out the important stuff.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.