Copy & paste cleanly into a command like grep, without terminal echo cluttering the output?

Question

I have some test output, looking like

PASS: tests/test_mutex_rmw
PASS: tests/test_mutex_trylock
PASS: tests/test_malloc_irreg
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
PASS: tests/test_groups
PASS: tests/test_group_split
PASS: tests/test_malloc_group
FAIL: tests/test_accs
FAIL: tests/test_accs_dla

I want to filter the output to just view the failures. It would be convenient to just copy the text from screen and paste into stdin to pass into grep, e.g.

grep FAIL

and Shift-Ctrl-V (or middle mouse button) to copy the text in.

What I want to see is just

FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
FAIL: tests/test_accs
FAIL: tests/test_accs_dla

But instead, the input pasted in is displayed to screen, and because of buffering the input gets mixed with the final output:

$ grep FAIL
PASS: tests/test_mutex_rmw
PASS: tests/test_mutex_trylock
PASS: tests/test_malloc_irreg
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
PASS: tests/test_groups
PASS: tests/test_group_split
PASS: tests/test_malloc_group
FAIL: teFAIL: tests/ARMCI_PutS_latency
sts/test_accs
FAIL: tests/test_accs_dla
FAIL: tests/ARMCI_AccS_latency
FAIL: tests/test_accs
FAIL: tests/test_accs_dla

It would make sense to me for the input to first be provided to cat and then passed to grep, cat | grep FAIL, but that doesn't actually help. The buffer mixup still occurs.

Of course it can be filtered cleanly if the data is placed in a file which is passed to grep. But what I'm looking for is a convenience tool to simply provide light filtering of text copied from the terminal output via the clipboard buffer. How is that best done?

Equivalently, how can pasting be done without echoing to screen (providing data silently as stdin for the command)?

One method is to explicitly switch off echoing,

stty -echo; grep FAIL; stty echo

That does work, but I suspect there are ways of doing it without toggling stty. Do you know other shell-based approaches?

I use bash (on Debian GNU/Linux), but POSIX shell solutions are also interesting.

Please edit your question and tell us what operating system you are using. — terdon, Commented Mar 9, 2022 at 12:23
bash on Linux, added to question. POSIX shell is also interesting. — Rizzer, Commented Mar 9, 2022 at 12:37
Is there some reason you can't pipe the output to grep when you execute the command that generates it? command | grep FAIL — Wastrel, Commented Mar 12, 2022 at 16:44
Several reasons. For one thing, I want all the output in general, that is I'll want to inspect other parts of the log in detail. Besides that, the full output actually is being written to file (via tee). Not convenient to inject the grep at the point where the log is actually generated. The question here is just for making it easier to quickly monitor progress and check parts of the log while the process is still running. Sometimes you're not expecting failure anyway, and want to filter what was supposed to be a routine output that you wouldn't have expected to need to inspect closely. — Rizzer, Commented Mar 13, 2022 at 1:48

Stéphane Chazelas · Accepted Answer · 2022-03-09 17:41:28Z

23

Mark the text with your mouse, then use xclip:

xclip -o | grep FAIL

or copied from Clipboard (Ctrl-c):

xclip -selection clipboard -o | grep FAIL

Or:

xclip -sel c -o | grep FAIL

for short.

edited Mar 9, 2022 at 17:41

Stéphane Chazelas

558k93 gold badges1.1k silver badges1.6k bronze badges

answered Mar 9, 2022 at 12:06

pLumo

22.7k2 gold badges42 silver badges68 bronze badges

Nice variant. I didn't have xclip installed, so it's a new tool for me. I'm hoping to find methods that react to Copy to give more control of the data transfer (the -selection clipboard option here "steals" anything that happens to be in the clipboard). I'll try playing with the other xclip options.
– Rizzer
Commented Mar 9, 2022 at 12:46
4

xsel is a very similar program you might look into a well
– frabjous
Commented Mar 9, 2022 at 23:56
3

(For the record, macOS — which is also Unix — has equivalent commands pbcopy and pbpaste.)
– gidds
Commented Mar 12, 2022 at 9:40

Add a comment |

Stéphane Chazelas · Accepted Answer · 2022-03-09 19:37:03Z

If you are using bash or any other Bourne-like shell, you can use a here-document. Begin your command with grep FAIL << 'EOF' and press Enter. Bash will prompt with a > symbol to show that more command input is expected from the terminal. Then paste in your input, and enter EOF by itself on a line. The output from grep will appear after the EOF line.

$ grep FAIL << 'EOF'
> PASS: tests/test_mutex_rmw
PASS: tests/test_mutex_trylock
PASS: tests/test_malloc_irreg
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
PASS: tests/test_groups
PASS: tests/test_group_split
PASS: tests/test_malloc_group
FAIL: tests/test_accs
FAIL: tests/test_accs_dla
EOF
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
FAIL: tests/test_accs
FAIL: tests/test_accs_dla

Make sure you quote the EOF (or any part of it), with any quoting operator, to make sure no parameter expansion, command substitution or arithmetic expansion is performed inside the here-document and \ characters be not mangled.

To minimise key strokes, you could make it:

grep FAIL<<\.
<paste-text-here>
.

Needs a few more keystrokes than the sponge method, but otherwise a useful alternative method, thanks. — Rizzer, Commented Mar 9, 2022 at 17:03

Chris Davies · Accepted Answer · 2022-03-15 21:23:21Z

16

You can use sponge to soak up all the stdin before writing it to stdout. (On Debian it's in the package moreutils.)

grep FAIL | sponge

or

sponge | grep FAIL

If you don't have sponge you can implement a simple approximation to it with POSIX code like this

#!/bin/sh
[ -n "$1" ] && exec 1>"$1"
umask 077
tmp="${TMPDIR:-/tmp}/${0##*/}.$$.tmp"
cat >"$tmp"
cat "$tmp"
rm -f "$tmp"

You mentioned you would really like sponge to have "an option to not echo the stdin" it doesn't echo stdin. What you're seeing is a function of the terminal driver being left at its default of echoing typed text. I'd be inclined to use a function such as this

esponge() {
    local g ss
    if [ -t 0 ]
    then
        g=$(stty -g)
        stty -echoe
    fi
    sponge "$@"
    ss=$?
    [ -n "$g" ] && stty "$g"
    return $ss
}

esponge | grep FAIL

Yet another alternative but equivalent to sponge is to send the input through tac (not POSIX but may be installed on some systems that don't have sponge)

tac | tac | grep FAIL

Or replace each instance of tac with one of these POSIX alternatives

edited Mar 15, 2022 at 21:23

answered Mar 9, 2022 at 13:01

Chris Davies

120k16 gold badges166 silver badges298 bronze badges

Excellent. sponge is pretty much exactly what I had in mind. Behaves like cat but separates stdin from stdout. The only way to perfect it further would be with an option to not echo the stdin, but I guess stty is needed to do that (or xclip with careful timing of clipboard contents)
– Rizzer
Commented Mar 9, 2022 at 13:53
1

Testing the sponge method some more, the pasted input can easily be visually separated from the filtered output simply by tapping enter a couple of times after pasting to get a gap of a few lines. Less need to avoid echoing the input in that case.
– Rizzer
Commented Mar 9, 2022 at 17:02
1

Note that sponge used to be a 5 line perl script before it was rewriten in C. That particular sponge usage can be written just perl -0777 -pe ''
– Stéphane Chazelas
Commented Mar 9, 2022 at 17:59
1

The advantage of something like this over xclip/xsel is that it can be used when the shell is on a remote system, e.g. via ssh/docker/qemu. Implementation can also be simpler for practical text-based cases, like sed ':b;N;bb' or printf '%s\n' "$(cat)".
– JoL
Commented Mar 10, 2022 at 2:43
Another quick-and-dirty way of simulating sponge (if you don't have the official version) is cat -n | sort -n | sed 's/ *[0-9]* //'
– Steve Summit
Commented Mar 11, 2022 at 21:55

Add a comment |

Stéphane Chazelas · Accepted Answer · 2022-03-11 11:43:28Z

While I'd go for the xclip/xsel approach that is by far the safest where applicable, in zsh, to make sure that whilst you paste text into grep, the terminal is not echoing back what you type, so that only the output of grep be seen, you can do:

STTY=-echo grep FAIL

and then paste your input, terminated with Ctrl+d (twice if the input doesn't end in a newline character).

From the manual:

STTY
If this parameter is set in a command's environment, the shell runs the stty command with the value of this parameter as arguments in order to set up the terminal before executing the command. The modes apply only to the command, and are reset when it finishes or is suspended. If the command is suspended and continued later with the fg or wait builtins it will see the modes specified by STTY, as if it were not suspended. This (intentionally) does not apply if the command is continued via 'kill -CONT'. STTY is ignored if the command is run in the background, or if it is in the environment of the shell but not explicitly assigned to in the input line. This avoids running stty at every external command by accidentally exporting it. Also note that STTY should not be used for window size specifications; these will not be local to the command.

You can achieve something similar in other POSIX-like shells (though here without handling suspensions) by defining a helper function such as:

noecho() (
  saved_tty_settings=$(stty -g)
  trap 'stty "$saved_tty_settings"' INT QUIT EXIT
  stty -echo
  "$@"
)

And then enter:

noecho grep FAIL

More generally, pasting text into a terminal is potentially dangerous even if you don't paste it as input to a shell. Here, if the text to be pasted has been copied from a terminal emulator in the first place, the risk is more limited though as terminal emulators won't store things like ^C / ^Z / ^D control characters in clipboard selections.

Controlling echo via an environment variable is a convenient method. Thanks. — Rizzer, Commented Mar 9, 2022 at 19:22
I think POSIX-ly $saved_tty_settings should not be double-quoted. POSIX allows stty -g to print "an unspecified form that can be used as arguments" (plural!); then it says "the form used shall not contain any characters that would require quoting to avoid word expansion by the shell". IMO this indicates that some implementations may require word splitting from unquoted $saved_tty_settings (i.e. they will fail when double-quoted) and no implementation shall require $saved_tty_settings to be double-quoted. — Kamil Maciorowski, Commented Mar 10, 2022 at 8:11
@KamilMaciorowski, that was a defect in the POSIX specification that will be fixed in the next release (on my request). No stty implementation has ever required the output of stty -g be split+glob'bed for reinput, that would not make sense. — Stéphane Chazelas, Commented Mar 10, 2022 at 8:44
Good to know, thank you. I can see you're involved in improving these things. Kudos. — Kamil Maciorowski, Commented Mar 10, 2022 at 8:54

Peter - Reinstate Monica · Accepted Answer · 2022-03-11 11:29:45Z

What's wrong with simply piping your pasted text through echo? At least for simple text like yours that works well enough. The only thing to remember is to quote the lines. So you write echo " , paste your text, close the quote and pipe it as desired. (This assumes, like in your example, that echodoes not corrupt your text. Control chars etc. may cause trouble with echo.)

$ echo "PASS: tests/test_mutex_trylock
PASS: tests/test_malloc_irreg
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
PASS: tests/test_mutex_trylock
PASS: tests/test_malloc_irreg
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
PASS: tests/test_groups
PASS: tests/test_group_split
PASS: tests/test_malloc_group
FAIL: tests/test_accs
"| grep FAIL
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
FAIL: tests/ARMCI_PutS_latency
FAIL: tests/ARMCI_AccS_latency
FAIL: tests/test_accs

Input and output are clearly separated. This method uses only the simplest *nix commands and is easier to remember for me than the syntax of a here document.

If you want to use the text as commands you can use echo as command substitution, for texts that are not too long.

I didn't know xclip, and msys2 doesn't seem to have it, so I'll continue to use my stupid command substitution method. I simply echo the buffer contents and make that the desired command's command line parameters.

Because the text consists of multiple line the pasted stuff (but not the echo, or the entire substitution!) it must be quoted.

Because the command substitution is not quoted the shell parses the echo output back into separate words.

The example below would of course be better solved with find ... -print0 (among others, file names with spaces don't work here) but it illustrates the point. (And anybody who puts spaces in paths deserves to be served a restraining order for computers anyway. Yes, Bill, I'm talking to you.)

$ ls -1 a*
a1
a2
a3
a4
a5
aa
$ ls -l $(echo "a1
a2
a3
a4
a5
aa
")
-rw-r--r-- 1 Peter None 0 Mar 11 11:59 a1
-rw-r--r-- 1 Peter None 0 Mar 11 11:59 a2
-rw-r--r-- 1 Peter None 0 Mar 11 11:59 a3
-rw-r--r-- 1 Peter None 0 Mar 11 11:59 a4
-rw-r--r-- 1 Peter None 0 Mar 11 11:59 a5
-rw-r--r-- 1 Peter None 0 Mar 11 11:58 aa

"anybody who puts spaces in paths deserves to be served a restraining order" There are three sorts of code, (1) throwaway code for specific instances where spaces may or may not break it, but you know and understand the input is safe, (2) code that will break if given unexpected spaces, (3) code that is sufficiently carefully written to cope with spaces even if they are not expected. Aim to write for (1) or (3) — Chris Davies, Commented Mar 11, 2022 at 11:28
@roima Spaces by no means break the OP's use case, as demonstrated. They may break file operations that I used as an example below (to illustrate an add-on that was not the original question) but do not, in the general case, break that method either. (And as you may know it is very hard to impossible to write shell code that is robust against pathological file names, so everything is <100%.) — Peter - Reinstate Monica, Commented Mar 11, 2022 at 11:33
As to the OP's requirements: I understood that the OP wants to avoid the terminal input mingling indistinguishably with the output (as is the case if you use the paste buffer simply as the terminal input, as in the OP's example). Using the text as explicit input (as with echo or a here document) seemed OK though, that is: The OP is not against echoing in general. This assumption is supported by the use case of parsing ASCII text (which echo handles just fine). — Peter - Reinstate Monica, Commented Mar 11, 2022 at 11:35
Thank you for clarifying. I had not used your suggestion in the way that you now demonstrate, and for me it had broken. Your example now clearly shows that it works +1 — Chris Davies, Commented Mar 11, 2022 at 11:59
Quoting via echo does work well enough. Has the obvious problem if there happens to be quotation characters in the text, but in many instances that won't be a problem. — Rizzer, Commented Mar 11, 2022 at 15:12

Zombo · Accepted Answer · 2022-03-11 14:32:17Z

3

For anyone using MSYS2, you can utilize /dev/clipboard:

grep FAIL /dev/clipboard

answered Mar 11, 2022 at 14:32

Zombo

1

Add a comment |

FeRD · Accepted Answer · 2022-03-12 11:17:52Z

2

Another moreutils solution with the advantage (or caveat, depending on your perspective) that it's a little more interactive, you can use vipe in a pipeline:

echo "" | vipe | grep FAIL

Your $EDITOR will launch, you can paste the text into it, even massage it however you see fit, and then save and close the editor. grep will be run on the final contents, without the input being shown in the terminal.

answered Mar 12, 2022 at 11:17

FeRD

9406 silver badges21 bronze badges

Add a comment |

Stack Exchange Network

Copy & paste cleanly into a command like grep, without terminal echo cluttering the output?

7 Answers 7

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
shell
grep
clipboard
.

Linked

Hot Network Questions

Copy & paste cleanly into a command like grep, without terminal echo cluttering the output?

7 Answers 7

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged shellgrepclipboard.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
shell
grep
clipboard
.