Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Access logs are more or less sorted by time, but to aggregate connections by time (uniq -c), you need to sort them a bit more. For a huge access log sort is very inefficient, because it buffers and sorts the whole file before printing out anything.

Do you know any option for sort or version of sort, that could sort only given ammount of lines at once, the print that block?

I have searched for the following keywords: "streaming sort", "block sort", "approximate sort". I have read the whole manual through, without use. Setting the buffer size (-S) did not influenced this.

share|improve this question
    
An example use: tail -f access_log | awk '{print $4, $1}' | sort --OPTION-IN-QUESTION | uniq -c". Thus you could see the number of requests by IPs from time to time. –  user39646 May 22 '13 at 9:17
add comment

4 Answers 4

up vote 2 down vote accepted

Try split --filter:

split --lines 1000 --filter 'sort ... | sed ... | uniq -c' access.log

This will split access.log into chunks of 1000 lines and pipe each chunk through the given filter.

If you want to save the results for each chunk separately, you can use $FILE in the filter command and possibly specify a prefix (default is x):

split --lines 1000 --filter '... | uniq -c >$FILE' access.log myanalysis-

This will generate a file myanalysis-aa containing the result of processing the first chunk, myanalysis-ab for the second chunk, etc.

The --filter option to split was introduced in GNU coreutils 8.13 (released in September 2011).

share|improve this answer
    
In CentOS 6 split does not have a filter option, though this seems to be a solution. Thanks! In the mean time, I had mine: tail -f log | perl -ne 'push(@buffer, $_); if ($n++==100) {print sort(@buffer); $n=0;}' –  user39646 May 22 '13 at 9:34
    
Seems plausible, the option is relatively new (added a line about it, above). –  chirlu May 22 '13 at 9:47
add comment
tail -f access_log | awk -v 'cmd=sort --OPTION-IN-QUESTION | uniq -c' '
    {print $4, $1 | cmd}
    NR % 1000 == 0 {close(cmd)}'
share|improve this answer
add comment

sort is not a functional panacea, you should use another appropriate tool to filter your data before it reaches sort at all. tail should probably do the job. For example, to get the last 100 lines sorted, you can do this:

tail -100 /var/log/foo.log | sort
share|improve this answer
1  
I know that. This is an answer to a different question. –  user39646 May 22 '13 at 9:12
    
@user39646 Then your question is unclear -- I don't see any other question. –  Chris Down May 22 '13 at 9:13
    
I have added a comment to make my question clearer. Here's an example scenario: tail -f log | sort --number-of-lines-to-sort-at-once=# –  user39646 May 22 '13 at 9:24
add comment

If you only want to sort part of the file you could use e.g. sed.

Sort line 15000 to 25000:

sed -n '15000,25000p' | sort

If you are going to keep the sorted files you could also consider split.

See man split.

share|improve this answer
    
I want to sort parts (plural!) of the log. For example: "tail -f access_log | awk '{print $4, $1}' | sort --OPTION-IN-QUESTION | uniq -c". Thus you could see the number of requests by IPs from time to time. Split pitily could output to files only. –  user39646 May 22 '13 at 9:13
    
You can sort multiple by that method. Add sed to loop where you increase range by wanted amount, or by split and all in one go. The use of tail -f makes it a completely different question. –  Sukminder May 25 '13 at 14:58
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.