1

Let's say I have a command foo that accepts a file and performs some transformation in that file (something similar to sed -i). And let's say that this command does not accept the classic "read from stdin, transform, write to stdout" option.

Is there some way to "transform" this command so that it allows using the stdin/stdout workflow?

The first thing I can think of is writing some kind of wrapper that uses temporary files. But I'd like to know if there is some kind of standardized "meta-tool" to do this (or some bash trickery).

  • You could use a FIFO (aka a Named Pipe). Write to that on one side, and read from and transform it on the other. The final output would go to a regular file. – DopeGhoti Nov 8 '17 at 19:03
  • In-place modification of streams is..... find another way to do it. – Ignacio Vazquez-Abrams Nov 8 '17 at 19:05
  • @DopeGhoti: You might be able to get that to work if foo (1) opens its file for reading and reads it entirely, (2) closes it, (3) opens it for writing (with truncation), and (4) writes to it.  Even so, as far as I understand it, your approach requires three processes to squabble over one named pipe, and that generally doesn’t work well unless they’re all written to share access and coordinate their actions. If the program opens the file for read/write and tries to seek, it will break. If, like sed -i, it creates a new file and renames it to the original name, your approach will fail. – G-Man Nov 8 '17 at 20:36
2

It’s a bit cheesy, but:

my_temp_file=$(mktemp)
cat > "$my_temp_file"
foo "$my_temp_file"
cat "$my_temp_file"
rm -f "$my_temp_file"

In case it isn’t obvious, this

  • reads stdin and writes it to a file,
  • invokes your foo program on the temporary file,
  • reads the temporary file and writes it to stdout, and
  • deletes the temporary file.

This will fail if the data to be processed are too big to fit into a file in /tmp.  You may be able to mitigate this a little by using the --tmpdir option to mktemp to put the temporary file in a filesystem that has more free space.

Connecting commands with && is all the rage these days.  It might make a little more sense to say

my_temp_file=$(mktemp) && {
    cat > "$my_temp_file"  && {
        foo "$my_temp_file"
        cat "$my_temp_file"
    }
    rm -f "$my_temp_file"
}

because

  • if mktemp fails, you don’t have a temporary file, and there’s nothing that you can do,
  • if cat > "$my_temp_file" fails, you haven’t captured the input, so there’s nothing that you can do (except you do still want to delete the temporary file)

If the exit status from foo is important to you, handle it appropriately.

Optionally make this a little more bullet-proof by setting up a trap to do the rm even if things go sideways.

1

Just like G-Man's answer, but encapsulated in a function:

foo_stream() { 
    local t=$(mktemp) && 
    cat - >| "$t" && 
    foo "$t" && 
    cat "$t" && 
    rm "$t"
}

So if foo is: foo() { sed -i 's/.*/& & &/' "$1"; } (to "triple" each line) then we use foo_stream in a pipeline any way we like:

$ seq 10 | foo_stream | tac
10 10 10
9 9 9
8 8 8
7 7 7
6 6 6
5 5 5
4 4 4
3 3 3
2 2 2
1 1 1

Note: I use set -o noclobber to avoid accidentally overwriting existing files, and >| redirection can override that.

  • 1
    May be splitting hairs, but if foo exits with a non-zero exit status or the second cat fails, those aren’t necessarily reasons not to delete the temporary file (unless there’s a requirement to keep it around for debugging purposes).  And you definitely might want to see the output even if foo exits with a non-zero exit status. – G-Man Nov 8 '17 at 20:54
  • Good point. I put this a community wiki, so edit away! – glenn jackman Nov 8 '17 at 23:00
0

If you have control over foo such that it does not edit in-place but you can provide an input file and an output file:

foo() { sed 's/.*/& & &/' "$1" > "$2"; }

Then you can use process substitutions to represent the "input pipe" and the "output pipe".

$ foo <(seq 10) >(tac)
10 10 10
9 9 9
8 8 8
7 7 7
6 6 6
5 5 5
4 4 4
3 3 3
2 2 2
1 1 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.