Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

When I use tee to pipe stdout directly to a "specific block of code" (which then writes the modified data to a file), I always get the full complement of exptected output lines in the file.

However, when I tee the same data to a process >("same block of code"), it can drop up to 4 lines from the output. This number varies; sometimes it writes all the lines.

How can I get around this issue, and why is it only happening in the >( process ) ?

More info: I am 'tee'-ing to 2 process-substitutions and also to normal stdout pipe when this line dropping occurs.

using GNU bash, version 4.1.5(1)-release (i486-pc-linux-gnu) on Ubuntu 10.04.2 LTS.

Here is the actual script. The second process-substitution is the one in question, the one beginning with >(tr $'\x60' $X01

#  
# Run 'locate' and direct the output to a temp file
  errflag=""
  locitmct=0  # Count of located items 
  columns=4  # The number of columns in the main dialog
  colmnb=0  # Column number (NB: columns 1 and 2 are processed together) 
  X01=$'\x01'
  eval locate $zenargs |tee \
      >(zenity --progress --pulsate --auto-close) \
      >(tr $'\x60' $X01 \
         |sed -n "s/^\(.*\/\)\(.*\)/\1\2\n\2\n\1/p" \
            |while IFS= read -r line ; do \
              #
              #
              # process the data
              #
              #
            done > "$listf" )\
      >/dev/null
#
cat "$listf"
#
share|improve this question
    
Your question is unclear. Can you provide a code sample? –  Mikel Apr 5 '11 at 6:47
    
Update to my question: I think(?) I've worked out why it is happening, but I have no idea of how to get around it.. It seems to be that when tee finishes sending it's last data, it exits and passes control to the following cat.. However my >(process) is still processing the last piece of data... I've tried wait but it has no effect. I suppose that is because process substitution is not a "normal" child process... btw, sleep 1 works, but that's just guesswork and what if it takes 10 seconds longer?.. there must be anoter way.. –  Peter.O Apr 5 '11 at 10:37
    
What shell is that? For once, it's important: bash, ksh and zsh don't implement process substitutions in exactly the same way. IIRC it's come up before on this site; unfortunately I can't find that thread (I guess it didn't have “process substitution” spelled out). –  Gilles Apr 5 '11 at 20:13
    
I'm using GNU bash, version 4.1.5(1)... (I've included it in the question body). –  Peter.O Apr 6 '11 at 6:05
add comment

1 Answer

up vote 2 down vote accepted

It has been 10 hours since I asked the question, and I've worked out a solution... Note: As I've previously mentioned in a comment (under the question), wait does not work with these processes. I assume that is because these 'process-substitutions' are not 'child' processes, which I believe is what wait waits for. (I tried wait with no args) ....

Comments on its features and failings would be appreciated. I don't quite understand how stdin is picked up by zenity and tr when echo is the first command, but I just thought I'd give it a go... It seems to work (in this case), but is this method safe?

There are quite likely to be some well tried and tested methods out there to tackle this, so other answers would be worthwile....


#!/bin/bash
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# Set up flag-files for processes to send PIDs to main process
# The first thing each process does is: echo -n "$BASHPID " > flag-file  
for i in {1..2};do cp /dev/null "$listf".pid$i;done
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
eval locate $zenargs |tee \
    >(echo -n "$BASHPID " > "$listf".pid1 ; \
      zenity --progress --pulsate --auto-close) \
    >(echo -n "$BASHPID " > "$listf".pid2 ; \
      tr $'\x60' $X01 \
       |sed -n "s/^\(.*\/\)\(.*\)/\1\2\n\2\n\1/p" \
          |while IFS= read -r line ; do \
              #
              #
              # process the data 
              #
              #
          done > "$listf" ) \
    >/dev/null
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
####  WAIT for processes to terminate  
pids=$(cat "$listf".pid{1,2})
while [[ "$pids" == *[0-9]* ]] ; do
   sleep .1 # GNU
   for pid in $pids ; do
      if ! kill -0 "$pid" 2>/dev/null; then
         pids="${pids/$pid/}"
      fi
   done 
done
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
cat "$listf"
#
share|improve this answer
    
'wait' is only for background processes started with '&' at the end of the line (at least in ksh, I would expect bash is the same). You might get better set's of eyes on this if you add your shell as a tag. Interesting problem, thanks for sharing ;-) ... Good luck! –  shellter Apr 5 '11 at 19:29
    
Do you know about the { print ${Hdr} ; cat ${file} ; } > outFile construct using { ; } pairs? (the semicolon as last token before closing brace is critical in in-line constructs). These can be quite complex, with conditional logic inside deciding is ultimately sent to > outFile. I hope this helps. (Sorry but I don't have the time to try and figure out exactly what your script is doing, I don't know anything about zenity). Also tag what OS you're using? –  shellter Apr 5 '11 at 19:35
    
@shellter: Thanks. I wasn't aware of the need for the final ;. In the context of my question, your { } example, gave a real insight into how ( ) background/substitution process and { } inline processes work in regards to std-I/O. It seems that they must open a FIFO pipe, which is available to any command anywhere in the process block. If I've got the right idea, it's really quite simple. tee must be opening a FIFO for each process. These processes may be fast or slow, and in my case, the tr process was slower than the inline stdout pipe, so cat only got part of the file. –  Peter.O Apr 6 '11 at 6:55
    
Re. improvements to the wait loop (above): ... If there is no native bash way to handle the 'wait' scenario, then I believe that aside from some other way of checking for a running process (rather than kill -0), then the only things it needs are.. 1) testing exit codes of each sub-process.. 2) checking that the flag-files actually exist, before proceeding. –  Peter.O Apr 6 '11 at 6:59
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.