0

I'm trying to get some trivial statistics on some debug output.

Each debug line is of the form(class name)(delimiter 1)(object ID)(delimiter 2)(method name)(delimiter 3)(log message)

I want to get a count of how many lines are coming from which methods.

Essentially, if each line can be reduced to (class name)(delimiter)(method name), I want to know how many occurences of each of those reductions appear in the log file.

What command can I run in Bash to do the counting?

(I'm doing this on macOS with macports replacing most of the default BSD-style tools with GNU tools.)

I can extract the class name with grep -o -E "^.*(delimiter 1), or extract the method name with grep -o -E "(delimiter 2).*(delimiter 3)", or highlight both with grep --color=always -E "^.*(delimiter 1)|(delimiter 2).*(delimiter 3)". I got stuck looking for a way to get grep to output just the two matches which could then be run through | uniq -c to do the counting.

Is there a way to get grep to print both matches for each line rather than just one match or the entire line?

  • 2
    Could you provide a sample of the expected output? – schrodigerscatcuriosity Mar 8 at 23:03
  • 2
    Also could you provide testable sample input data? – schrodigerscatcuriosity Mar 8 at 23:33
  • You mean grep -Eo 'delimiter (1|2)' or grep -Eo 'delimiter (1|3)' does not give you the result you wanted? – Jetchisel Mar 10 at 4:28
  • @Jetchisel: Correct, it does not. grep -o -E "^.*(delimiter 1)|(delimiter 2).*(delimiter 3)" extracts the class name and method name onto separate lines. Worse, it finds matches for one or the other (but not both) in some of the regular output, so even merging pairs of lines of that would not work. – ShadSterling Mar 10 at 22:28
0

In essence, it can be done with

sed -r -n 's/(^.*)(delimiter 1)(.*)(delimiter 2)(.*)(delimiter 3)(.+$)/\1(delimiter)\5/p' <( command that generates debug logs ) | sort | uniq -c | sort -rn

(adapted from here)

  • .* may match too much; sed is greedy and wants to match as much as possible as early as possible, so those may need to be e.g. negations of the delimiters (which can be complicated if you have inconvenient delimiters)
  • Going from ^ to $ is important, if your expression doesn't match the entire line sed will include the unmatched portion in the output
  • Parenthesis are only necessary around the class name and method name; removing the others means changing the numbers at the end, because the numbers refer to parenthesized subexpressions in order. (Including them all makes it possible to show more of what's going on in the sed output, e.g. by changing the end to /\1(delimiter)\5 -- \1\2\3\4\5\6\7/p)
  • sort has to be run before uniq -c because uniq -c only counts runs of consecutive identical lines, non-consecutive identical lines get separate counts
  • uniq -c can't be replaced by sort -u because sort -u only drops duplicates it doesn't count them
  • The final sort isn't necessary to answer the question as asked
  • Yes, if you use regular expressions to solve a problem, now you've got two problems.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.