Use uniq to filter adjacent lines in pipeline

Question

I'm trying to monitor theme changes using this command:

dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ."

Output right now looks like this:

uint32 0
uint32 0
uint32 1
uint32 1
uint32 0
uint32 0
uint32 1
uint32 1

This output comes from theme toggling. The theme notification shows up twice for some reason. Now I want to pipe it to uniq so I only remain with a single entry like so:

uint32 0
uint32 1
uint32 0
uint32 1

However appending uniq at the end does not produce any output anymore.

dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ." | uniq

From man uniq:

Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).

uniq needs to buffer at least the last output line to be able to detect adjacent lines, I don't see any reason why it could not buffer it and pass it along the pipeline. I've tried tweaking line buffering as suggested here but the results are still the same for me.

dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ." | stdbuf -oL -i0 uniq

Answer 1

It's the usual behavior of many tools, inherited from default Standard C library behavior with functions manipulating I/O streams (fopen(3), fwrite(3) ...). This is documented in setvbuf(3) (or a deprecated variant). Several *nixes tell the same: *BSD, Solaris, GNU/Linux..., but POSIX doesn't specify it and ISO/IEC 9899 is too elusive. GNU tells:

Newly opened streams are normally fully buffered, with one exception: a stream connected to an interactive device such as a terminal is initially line buffered.

(Also, stderr is usually unbuffered in all the implementations. It doesn't matter for this Q/A.)

So for most text filtering commands (not all, for example cat doesn't have this behavior), whenever it's not the last command in a pipeline, its output is not line buffered anymore, and a linefeed will not trigger the immediate emission of data for the next command to process.

---

Here the relevant command is not uniq but grep because its output switches from terminal to non-terminal. GNU grep has an option to change this behavior: --line-buffered:

--line-buffered

Use line buffering on output. This can cause a performance penalty.

If the command doesn't have a specific option to choose the behavior, one can still use the command stdbuf which alters behavior by using setvbuf(3) through LD_PRELOAD mechanism. To restore line-buffered behavior for a command that is not at the end of the pipeline, one can prefix this command with stdbuf -oL .

For OP's case:

dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" |
    grep --line-buffered -o "uint32 ." | uniq

or if this grep command didn't also have a specific option:

dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" |
    stdbuf -oL grep -o "uint32 ." | uniq

Note that in both cases, uniq being at the end of the pipeline didn't need any adjustment to its default behavior.

Should later the pipeline be augmented and uniq not be at the end anymore and thus not outputting to a terminal anymore, it would also change behavior and would require the same treatment. For example the line-buffered behavior of the uniq command alone on a terminal:

uniq

changes to full-buffered with :

uniq | cat

but can be reversed to line-buffered with:

stdbuf -oL uniq | cat