7

I frequently want to do something like this:

 cat file | command > file

(which obviously doesn't work). The only solution I've seen for this is sponge, i.e.

 cat file | command | sponge file

Unfortunately, sponge is not available to me (nor can I install it or any other package).

Is there a more standard quick way to do this without having to break it up every time into multiple commands (pipe to temp file, pipe back to original, delete temp file)? I tried tee for example, and it seems to work, but is it a consistent/safe solution?

3
  • 2
    If there were a more standard way, sponge wouldn't have had to exist. If you can't install a package, can you create the corresponding shell script to use? (x=$(mktemp); cat > "$x" ; mv "$x" "$1") Sep 24 '19 at 23:09
  • 1
    Yeah, that would work. Why is tee not a viable solution?
    – argentum2f
    Sep 25 '19 at 13:54
  • 1
    sponge is itself just creating a temporary file and moving it onto the original (and breaking hardlinks), so better be honest about it (like in @MichaelHomer's example), rather than hide the ugly truth under the rug, and pretend that it just does something magical ;-) An improvement imho to that would be (x=$(mktemp); cat >"$x" && mv "$x" "$1" || rm "$x").
    – mosvy
    Feb 5 '20 at 8:41
6

A shell function replacing sponge:

mysponge () (
    append=false

    while getopts 'a' opt; do
        case $opt in
            a) append=true ;;
            *) echo error; exit 1
        esac
    done
    shift "$(( OPTIND - 1 ))"

    outfile=$1

    tmpfile=$(mktemp "$(dirname "$outfile")/tmp-sponge.XXXXXXXX") &&
    cat >"$tmpfile" &&
    if "$append"; then
        cat "$tmpfile" >>"$outfile"
    else
        if [ -f "$outfile" ]; then
            chmod --reference="$outfile" "$tmpfile"
        fi
        if [ -f "$outfile" ]; then
            mv "$tmpfile" "$outfile"
        elif [ -n "$outfile" ] && [ ! -e "$outfile" ]; then
            cat "$tmpfile" >"$outfile"
        else
            cat "$tmpfile"
        fi
    fi &&
    rm -f "$tmpfile"
)

This mysponge shell function passes all data available on standard input on to a temporary file.

When all data has been redirected to the temporary file, the collected data is copied to the file named by the function's argument. If data is not to be appended to the file (i.e -a is not used), and if the given output filename refers to an existing regular file, if it does not exist, then this is done with mv (in the case that the file is an existing regular file, an attempt is made to transfer the file modes to the temporary file using GNU chmod first). If the output is to something that is not a regular file (a named pipe, standard output etc.), the data is outputted with cat.

If no file was given on the command line, the collected data is sent to standard output.

At the end, the temporary file is removed.

Each step in the function relies on the successful completion of the previous step. No attempt is made to remove the temporary file if one command fails (it may contain important data).

If the named file does not exist, then it will be created with the user's default permissions etc., and the data arriving from standard input will be written to it.

The mktemp utility is not standard, but it is commonly available.

The above function mimics the behaviour described in the manual for sponge from the moreutils package on Debian.


Using tee in place of sponge would not be a viable option. You say that you've tried it and it seemed to work for you. It may work and it may not. It relies on the timing of when the commands in the pipeline are started (they are started pretty much concurrently), and the size of the input data file.

The following is an example showing a situation where using tee would not work.

The original file is 200000 bytes, but after the pipeline, it's truncated to 32 KiB (which could well correspond to some buffer size on my system).

$ yes | head -n 100000 >hello
$ ls -l hello
-rw-r--r--  1 kk  wheel  200000 Jan 10 09:45 hello
$ cat hello | tee hello >/dev/null
$ ls -l hello
-rw-r--r--  1 kk  wheel  32768 Jan 10 09:46 hello
6
  • @JdeBP It does, I'll update.
    – Kusalananda
    Jan 10 '20 at 11:06
  • Unlike the real sponge, your function will leave the original file in a messed up state if the cat fails midway. That's a total FAIL, sorry. If atomicity && data integrity don't matter, but preserving the original inode is essential, there are smarter and faster ways to do it than copying the whole file once (like sponge) or twice (like your function).
    – mosvy
    Feb 5 '20 at 9:05
  • @mosvy Are you talking about the first cat, to $tmpfile? If that cat fails, the original file will not be messed up. Could you possibly explain what types of failure conditions you're seeing and how I may improve the code to cope with these?
    – Kusalananda
    Feb 5 '20 at 9:25
  • No, I'm talking about the second: cat "$tmpfile" >"$outfile". What's the user supposed to do if that fails midway? Go hunt through the temporary files (wherever her OS may hide them until she finds her precious data?
    – mosvy
    Feb 5 '20 at 9:28
0

There's this short bash script, which requires Perl
https://github.com/ildar-shaimordanov/perl-utils#sponge

The second script should be a drop-in replacement for the version in moreutils

There's also a version that is a stand-alone perl script.

1
  • 1
    Note that the in-line Perl script employed in the linked-to code will buffer all data in memory. In situations where the data is huge, this may not be a viable option.
    – Kusalananda
    Jan 10 '20 at 9:36
0
function wf() {
    #create a temporary file
    local tmpf="${1}_$(< /dev/urandom tr -dc A-Za-z0-9 | head -c16)"
    #redirect the result
    cat > $tmpf
    #replace the original file
    mv -f $tmpf "${1}"
}

next we use the function

grep "error" messages.log | wf messages.log

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.