Redirect output to file but never create empty file

Question

Is there a way to redirect output of a command that never stops to a file but only create the file once the command actually produces output?

Here is what I have so far:

target=logs/$(date +%Y%m%dT%H%M%S).mqtt
mosquitto_sub -v -t "#" --qos 0 >> $target

But this produces an empty file if no messages are sent to mosquitto (hence mosquitto_sub produces no output). I don't want that empty file.

I have two constraints:

1. my system experiences sudden power cuts during normal operation and
2. the log files are synced across the network all the time.

This is on a Raspberry Pi 4.

Therefore I need a solution that

1. never creates an empty file (deleting it later is not an option) and
2. starts writing the file without significant delay after mosquitto_sub starts writing to stdout.

Edit

Combining the solutions suggested by both duplicate targets still results in buffering:

stdbuf -oL -eL mosquitto_sub -v -t "#" --qos 0 | ifne tee $target

The file $target is created as soon as mosquitto_sub results in output (i.e. as soon as a message is sent to mosquitto with mosquitto_pub) but it remains empty until the buffer fills up, then lots of messages appear at once. Such a system would risk too much data loss in case of a power cut.

Applying stdbuf to ifne doesn't help either. The alternative unbuffer doesn't even install successfully on my test system which is Linux Mint.

Answer 1

Yes, ifne is not sensitive to stdbuf because even though it is using stdio (whose buffering stdbuf is meant to adjust), it's not printing its data to stdout, but to a different FILE* resulting of fdopen() on a pipe (to the command it executes).

You could always emulate ifne with zsh:

stdbuf -o0 cmd | zsh +o multibyte -c '
  read -k -u0 c && {
    print -rn -- $c && exec cat
  } >> logs/${(%):-%D{%Y%m%dT%H%M%S}}.mqtt'

Using zsh rather than any other shell makes it easier to deal with arbitrary output as other shells can't deal with NUL bytes, so you'd have to resort to cumbersome encoding to work around it.

I've never come across an implementation of cat that buffers its output, but standard cat do support a -u option to unbuffer it, so you might find some implementations where that's needed.

You may also want to set the pipefail option (supported by both zsh and bash) so as to be able to report cmd's failures in addition to and empty output and the failures to open the output file or cat's failure to read or write if any.

Above the timestamp in the output file name is generated using the %D{format} prompt expansion operator (prompt expansion enabled upon parameter expansion with the % parameter expansion flag), it is done after the first byte of the output of cmd has been read, you may want to move it to a output=logs/${(%):-%D{%Y%m%dT%H%M%S}}.mqtt at the start of that inline zsh script for that timestamp to correspond to when cmd was started rather than when it started to output something.

If making your script a zsh script instead of a bash script, you can also skip the zsh -c and do:

#! /bin/zsh -
output=logs/${(%):-%D{%Y%m%dT%H%M%S}}.mqtt
set -o pipefail
stdbuf -o0 cmd | (
  set +o multibyte
  read -k -u0 c && {
    print -rn -- $c && exec cat
  } >> $output
)