Access logs are more or less sorted by time, but to aggregate connections by time (uniq -c), you need to sort them a bit more. For a huge access log sort is very inefficient, because it buffers and sorts the whole file before printing out anything.

Do you know any option for sort or version of sort, that could sort only given ammount of lines at once, the print that block?

I have searched for the following keywords: "streaming sort", "block sort", "approximate sort". I have read the whole manual through, without use. Setting the buffer size (-S) did not influenced this.

  • An example use: tail -f access_log | awk '{print $4, $1}' | sort --OPTION-IN-QUESTION | uniq -c". Thus you could see the number of requests by IPs from time to time. – user39646 May 22 '13 at 9:17

Try split --filter:

split --lines 1000 --filter 'sort ... | sed ... | uniq -c' access.log

This will split access.log into chunks of 1000 lines and pipe each chunk through the given filter.

If you want to save the results for each chunk separately, you can use $FILE in the filter command and possibly specify a prefix (default is x):

split --lines 1000 --filter '... | uniq -c >$FILE' access.log myanalysis-

This will generate a file myanalysis-aa containing the result of processing the first chunk, myanalysis-ab for the second chunk, etc.

The --filter option to split was introduced in GNU coreutils 8.13 (released in September 2011).

  • In CentOS 6 split does not have a filter option, though this seems to be a solution. Thanks! In the mean time, I had mine: tail -f log | perl -ne 'push(@buffer, $_); if ($n++==100) {print sort(@buffer); $n=0;}' – user39646 May 22 '13 at 9:34
  • Seems plausible, the option is relatively new (added a line about it, above). – chirlu May 22 '13 at 9:47
tail -f access_log | awk -v 'cmd=sort --OPTION-IN-QUESTION | uniq -c' '
    {print $4, $1 | cmd}
    NR % 1000 == 0 {close(cmd)}'

sort is not a functional panacea, you should use another appropriate tool to filter your data before it reaches sort at all. tail should probably do the job. For example, to get the last 100 lines sorted, you can do this:

tail -100 /var/log/foo.log | sort
  • 1
    I know that. This is an answer to a different question. – user39646 May 22 '13 at 9:12
  • @user39646 Then your question is unclear -- I don't see any other question. – Chris Down May 22 '13 at 9:13
  • I have added a comment to make my question clearer. Here's an example scenario: tail -f log | sort --number-of-lines-to-sort-at-once=# – user39646 May 22 '13 at 9:24

If you only want to sort part of the file you could use e.g. sed.

Sort line 15000 to 25000:

sed -n '15000,25000p' | sort

If you are going to keep the sorted files you could also consider split.

See man split.

  • I want to sort parts (plural!) of the log. For example: "tail -f access_log | awk '{print $4, $1}' | sort --OPTION-IN-QUESTION | uniq -c". Thus you could see the number of requests by IPs from time to time. Split pitily could output to files only. – user39646 May 22 '13 at 9:13
  • You can sort multiple by that method. Add sed to loop where you increase range by wanted amount, or by split and all in one go. The use of tail -f makes it a completely different question. – Runium May 25 '13 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.