Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Given a command the output of which is being processed by, for example, grep I also want the actual output to be included in the log file for debugging purposes.

For example if I have the following

useful=$(curl -s http://example.com/some/data | grep 'useful line')
echo "useful=$useful"

I want to see

This page has a number of lines in it, but'
useful lines are the only ones I care about
except when something goes wrong and then
I'd really like to see what the heck was in
the output that grep consumed.
useful=useful lines are the only ones I care about

This can be done with tee like so

useful=$(curl -s http://example.com/some/data | tee /proc/$$/fd/1 | grep 'useful line')
echo "useful=$useful"

but if stdout is being redirected to a file tee clobbers the rest of your log file. tee -a fails in largely the same way.

share|improve this question
2  
Please explain what you mean with redirecting stdout to a file. Post the exact command which fails, explain how it fails and what you expect it to do instead. –  lesmana Aug 27 '11 at 18:51
    
Sorry, I tried to be as clear as possible without making the example too complicated. The goal is to be able to find out what went wrong when the grep fails to get what it was supposed to out of the command. –  ajorg Aug 29 '11 at 17:25
add comment

5 Answers

tee the stdout stream to the controlling terminal device /dev/tty.

(
exec 1> >(tee -a stdout.log)
: > stdout.log
#var="$(echo -e "one\ntwo" | tee /dev/tty | grep one)"
var="$(echo -e "one\ntwo" | tee -a /dev/tty stdout.log | grep one)"
echo "var=$var"
)

cat stdout.log
# one
# two
# var=one
share|improve this answer
    
Using a tty is a neat idea but unfortunately I want the data to end up in the log file, not on the console. –  ajorg Aug 29 '11 at 17:31
add comment

I'm not clear whether you want $var, or is $var just a means to an end. To get $var, add var="$(cat log)"

sed -nre 'p;/one/w log' <(echo -e "one\ntwo")
  • -------(stdout)
    one
    two
  • -------(log)
    one

Or do you mean something like this:

sed -nre 'p;/one/w log' -e 's/one/logged: &/p' <(echo -e "one\ntwo")
  • -------(stdout)
    one
    logged: one
    two
  • -------(log)
    one

Note: sed's w command overwrites log each time the script is executed. You can, of course, append log to a main log after sed runs.  See: Command Summary for sed

share|improve this answer
add comment

Why not just use a function:

#!/bin/bash
logAndEcho () {
    echo -e "$@" >> LOGFILE.txt
    echo -e "$@"
}
var=$(logAndEcho "one\ntwo\nThis is a test"|grep one)
echo "var=$var"

This will send the entire results to the logfile and then echo the data to stdin where you can capture it as you need.

share|improve this answer
add comment

I think you have to use an additional file descriptor and tee the stdout stream to it so that the duplicated stdout stream can get redirected to a log file. This way the normal stdout stream via /proc/$$/fd/1 is left unchanged. And if grep fails to match a line, you should have the entire command output that grep consumed available in your log file for debugging purposes.

useful="$(
exec 3<&1 3>stdout.log
curl -s http://example.com/some/data | tee /proc/$$/fd/3 | grep 'useful line'
exec 3<&-
)"
echo "useful=$useful"


# test case
(
# dup stdout using tee & an additional file descriptor
# open fd 3 for reading from where fd 1 is pointing to (i.e. stdout)
# anything that is being redirected to fd 3 will be redirected to stdout.log 
#exec 3<&1 3>stdout.log
exec 3<&1 3> >(tr '[:lower:]' '[:upper:]' > stdout.log)
echo -e "one\ntwo" | tee /dev/fd/3 | grep one        # Mac OS X
#echo -e "one\ntwo" | tee /proc/$$/fd/3 | grep one   # Linux
exec 3<&-
) 

Another approach might be to use named pipes or Bash Co-Processes (available in Bash 4.0 and above).

For more information on file descriptors see: File Descriptor 101

share|improve this answer
add comment

sed is designed for this - it's the stream editor. It can replace tee and grep in your process, or be combined with tee for a few more tines in your fork, if you like. Anyway, see for yourself:

% sed '/useful/w ./useful.log' <<HDOC >>./notsomuch.log
> maybe useful
> definitely not
> ho-hum
> now this is useful
> HDOC
% cat ./useful.log
OUT> maybe useful
OUT> now this is useful
% cat ./notsomuch.log
OUT> maybe useful
OUT> definitely not
OUT> ho-hum
OUT> now this is useful
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.