5

I'm looking for a "meta-command" xyz such that:

(echo "foo"; echo "bar") | xyz rev

will return:

foo oof
bar rab

I'd like to avoid temporary files, i.e. I'm looking for a solution neater than:

tempfile=$(mktemp)
cat > $tempfile
cat $tempfile | rev | paste $tempfile -

(And of course I want a general solution, for any command, not just rev, you can assume that the command outputs exactly one line for each input line.)

Zsh solution would be also acceptable.

6

There will be a lot of problems in most cases due to the way that stdio buffering works. A work around for linux might be to use the stdbuf program and run the command with coproc, so you can explicitly control the interleaving of the output.

The following assumes that the command will output one line after each line of input.

#!/bin/bash
coproc stdbuf -i0 -o0 "$@"
IFS=
while read -r in ; do
    printf "%s " "$in"
    printf "%s\n" "$in" >&${COPROC[1]}
    read -r out <&${COPROC[0]}
    printf "%s\n" "$out"
done

If a more general solution is needed as the OP only required each line of input to the program to eventually output one line rather than immediately, then a more complicated approach is needed. Create an event loop using read -t 0 to try and read from stdin and the co-process, if you have the same "line number" for both then output, otherwise store the line. To avoid using 100% of the cpu if in any round of the event loop neither was ready, then introduce a small delay before running the event loop again. There are additional complications if the process outputs partial lines, these need to be buffered.

If this more general solution is needed, I would write this using expect as it already has good support for handling pattern matching on multiple input streams. However this is not a bash/zsh solution.

  • This approach runs the command once, which the OP requests in followup comments. This could be important if the command being run was say awk '{s[$0]++; print s[$0], " ", $0}' to print out how many times this line has been seen already. – icarus Sep 15 at 7:06
  • I thought about doing something like this with Perl (or awk if it can do that). But I think any solution like this would also depend on the filter reading and writing in a nicely synchronized fashion. If it wants to read the whole file before outputting anything, this would seem to hang. Same if it reads a couple of lines ahead. I wonder if there is any sensible way around using temp files. – ilkkachu Sep 15 at 11:43
  • @ilkkachu I agree entirely. I am relying on the OP's statement "you can assume that the command outputs exactly one line for each input line" and stretching it to mean that the command outputs one line after each is read. If I was actually wanting to do this in practice I would use expect as I find it a good scripting language with easy support for handling multiple input streams. Perl would be fine, and has the expect.pm module. I don't think awk can do it. – icarus Sep 15 at 18:50
  • yeah, I took it as referring to just the number of output lines (and then the corner cases started creeping into mind). Now that you mention it, it can be read in the stricter sense too. In any case, it's good to be explicit about any limitations, so that no-one gets caught at them by surprise. – ilkkachu Sep 15 at 19:01
  • @ilkkachu Added a note on the assumptions and some comments how to do a more general solution in shell, not that I think it is "sensible". The OP didn't want to use temp files. – icarus Sep 15 at 19:28
5

A shell function. This will work in any shell that supports <<< here-strings, including zsh and bash.

xyz() { 
    while read line
    do 
        printf "%s " "$line"
        "$@" <<<"$line"
    done
}

$ (echo "foo"; echo "bar") | xyz rev
foo oof
bar rab
  • 1
    @JL2210 you're going to get surprising results when $line is equal to something like -e or \n (not newline but literal backslash-en) – Oh My Goodness Sep 14 at 19:20
  • 2
    @JL2210: Not in POSIX. In ash for example you'd get: -nE a. I know that this question is about bash and zsh but I agree that printf is much more portable and should be used here, or pretty much anywhere instead of echo. – Arkadiusz Drabczyk Sep 14 at 19:27
  • 3
    @OhMyGoodness, If I read this right, this launches a new copy of rev or whetever for each line, so it will only work with commands that only look at the current line when producing output. But not something where the absolute line number or the contents of the surrounding lines affect the output. – ilkkachu Sep 15 at 7:41
  • 4
    The user said they would want to avoid using temporary files. Using a here-string would make bash create a temporary file containing the string. – Kusalananda Sep 15 at 8:38
  • 1
    @OhMyGoodness You should include the -f option to strace see what the cat process is actually reading. On an Ubuntu system, the shell creates a temporary file under /tmp, writes foo to the file descriptor, closes it, opens it again, unlinks it, and passes the fd of the opened file to cat, which reads it. The same happens on OpenBSD. – Kusalananda Sep 15 at 22:07
2

In this particular case, I'd use perl:

printf '%s\n' foo bar | perl -Mopen=locale -lpe '$_ .= " " . reverse$_'
foo oof
bar rab

Which you could extend to also work with grapheme clusters:

$ printf '%s\n' $'complique\u301' |
    perl -Mopen=locale -lpe '$_ .= " " . join "", reverse/\X/g'
compliqué éuqilpmoc

Or zsh:

while IFS= read -r line; do
  print -r -- $line ${(j::)${(s::Oa)line}}
done

Though I would avoid using while read loops to process text, even in zsh (even if in this particular case, it's not so bad as only builtin commands are used).

In the general case, using temp files is probably the best approach. With zsh, you can do it with:

(){paste -d ' ' $1 <(rev <$1)} =(print -l foo bar)

(where =(...) takes care of creating and cleaning up the temp file).

Replacing it with pipes and some form of teeing is a recipe for deadlock in the general case. See these similar questions for some approaches and details about the deadlock situations:

1

This is a gawk implementation of the co-process approach outlined by icarus and therefore subject to the same stdio buffering considerations, also see this

{ echo foo; echo bar; } |  gawk -vcmd=rev '
 BEGIN {c=sprintf("stdbuf -i0 -o0 \"%s\"", cmd)}
 {
  printf "%s ", $0
  print |& c; fflush()
  c |& getline
  print
 }'

foo oof
bar rab
0

Named pipes to the rescue!

xyz() {
  pipe="/tmp/$$pipe"
  mkfifo "$pipe"
  tee "$pipe" | "$@" | paste "$pipe" -
  rm "$pipe"
}

Results on your example command:

$ (echo "foo"; echo "bar") | xyz rev
foo oof
bar rab

The above function does exactly what you describe.

  • 1
    That's probably the kind of approach the OP had in mind but note the potential for deadlock though. Like with seq 100000 | xyz sed '1!d'. You'd want to avoid /tmp or any publicly accessible area if you're going to use predictable names and potentially world readable fifos (depends on umask here) – Stéphane Chazelas Sep 16 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.