Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using gradle run to start a REST server. The output of the REST server looks like this:

XXX.XXX.XX.XXX - <moreinfo>
randomtext
randomtext
XXX.XXX.XX.XXX - <moreinfo>
XXX.XXX.XX.XXX - <moreinfo>
randomtext
XXX.XXX.XX.XXX - <moreinfo>

XXX.XXX.XX.XXX here is an IP address, randomtext are error messages. All output is directed to stdout, sadly.

How can I direct all lines starting with an IP address to a file called err.log and every other line to all.log?

Unfortunately, gradle run can only be started once and doesn't stop, since it is a REST server.

Maybe use a tee , grep combination?

share|improve this question

2 Answers 2

up vote 7 down vote accepted

In Bash, you can use process substitution with tee:

tee >(grep XXX > err.log) | grep -v XXX > all.log

This will put all lines matching XXX into err.log, and all lines into all.log. >( ... ) creates the process in the parentheses and connects its standard output to a pipe. This works in zsh and other modern shells too.

You can also use the pee command from moreutils:

pee "grep XXX > err.log" "grep -v XXX > all.log"

pee redirects standard input to multiple commands ("tee for pipes").

A further alternative is with awk:

awk '{ if (/^([0-9]{1,3}\.){3}[0-9]{1,3}/) { print > "err.log" } else { print > "all.log" } }'

That just tests every line against the expression and writes the whole thing into err.log if it matches and all.log if it doesn't.

The awk regular expression is suitable for grep -E too (although it does match some bad addresses — 999.0.0.0 and so on — but that probably isn't a problem).

share|improve this answer
    
Hmm err.log is empty and all output is redirected to all.log using the tee command above. –  polym Jul 6 at 22:18
    
Check that your regular expression is actually matching the right lines — if err.log exists at all then the command ran but nothing came out. grep -E with the expression used in the awk command ought to match, or it does here. –  Michael Homer Jul 6 at 22:23
    
Ah ok I got it. Can you modify your question so that all.log doesn't contain the matched lines in the grep expression? –  polym Jul 6 at 22:26
    
Done — wasn't sure which one you wanted, so I had both. –  Michael Homer Jul 6 at 22:28
    
Oh darn sorry, it worked. I overwrote all.log and err.log with an old command. Sorry for the confusion. Thanks, you're awesome :))!! –  polym Jul 6 at 22:36

So, it looks like gradle run doesn't comply with tee, pee, grep and io-redirection. It always stops reading after 4096 bytes.

To circumvent this issue, I read each line of gradle run. I didn't test it yet, but I guess that reading a line that is over 4k characters long will also fail.

Anyway, here is the code to solve my question specifically:

#!/bin/bash
STDOUTLOG="/log/stdout.txt"
STDERRLOG="/log/stderr.txt"
while read -r line; do
    [[ $line =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}.* ]] && printf '%s\n' "$line" >> "$STDERRLOG" && continue
    printf '%s\n' "$line" >> "$STDOUTLOG"
done < <(gradle run)
share|improve this answer
1  
You should use read -r line and printf '%s\n' "$line" to avoid some edge cases breaking things. –  nyuszika7h Jul 7 at 13:07
    
@nyuszika7h Thank you! I modified the answer accordingly :). –  polym Jul 7 at 20:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.