0

I frequently need to run a script and provide it several lines of STDIN. When doing this, I'll typically prepare the invocations, and then copy & paste them from one terminal window to another. Ideally, I'd like to be able to paste blocks of text such as:

<script name> <script args>
<STDIN line 1>
<STDIN line 2>
...

after which I'll hit control-d to indicate to the script that STDIN has reached the end of input.

If I do this in bash (as a single paste), the script executes, none of the STDIN echos in the session, an extra <cr> must be hit before ^d is handled, and the first line of the script's STDIN is lost. So, I typically paste the script invocation as its own paste, then all the lines of STDIN as a second paste. Doing it this way, the lines pasted echo, and I can hit control-d to indicate end of input.

If I do this under zsh, most of the STDIN lines don't echo (if there are a lot of them, some of the last lines will echo, the first of which may only echo a trailing fragment of the line), and again, an extra <cr> is needed before ^d is recognized.

It doesn't matter if the script in question is python or perl...both behave the same, which leads me to believe this is a shell issue.

The question is, what's the issue, and is there a fix that would allow doing each execution as a single paste?

The content of the script being run really doesn't matter. In testing, I've used scripts as simple as a while loop reading STDIN line by line into a buffer, then outside the while loop, print the buffer. By <cr>, I mean the enter or return key...depending on what it is on your keyboard.

2
  • 1
    Can you please edit your question and add a simple, minimal (python or perl as you prefer) script that reproduces the error so that we can understand and also use it for testing? Are you sure you mean <cr> by the way? Don't you mean newline?
    – terdon
    May 17 at 18:15
  • I can't reproduce this problem with bash under either Cygwin mintty or xterm
    – roaima
    May 17 at 21:36
3

Try using a heredoc, they're pretty easy to cut and paste:

script.sh arg arg arg <<'END_INPUT'
line1
line2
line3
END_INPUT
3
  • That's a workaround :) And it actually does work and solves another issue, which is that sometimes when doing this, I'm actually doing it several times with different args and different input. Done as a series of heredocs, I can paste several at once, and each gets its own "end of input", does its thing, terminates, and then the next can run. But I'm still curious what's going on the original way. May 17 at 18:40
  • Why not just add a trailing \ at the end of each line until you get to the last line then when you press enter the command will be executed May 17 at 20:06
  • @JonathanLewis What's going on "the original way" is that you're pasting three separate lines like cmd arg1 arg2 followed by line of stdin and other line of stdin which all gets interpreted by your shell as input to your shell, NOT as standard input to cmd arg1 arg2. It's much the same as if you had pasted in, all in one line: cmd arg1 arg2; line of stdin; other line of stdin
    – Jim L.
    May 18 at 21:20
0

Most likely, what you are observing depends on line editing and the mode your shell sets the terminal in.

When you paste the text

cat
a
b

(including a newline after "b") to the command line of an interactive Bash session, by default it finds the terminal set in raw mode — as it is when line editing is provided by the GNU Readline library; it means that, among other things, input is sent to the shell character by character (not line by line), the conversion of carriage return characters (\r, what the terminal sees when Enter is pressed or a newline is pasted to the command line) into line feed characters (\n) is disabled and typed characters are echoed to the command line by the shell itself (not by the terminal).
The shell then reads the first line, finds it is a complete simple command (cat), and runs it. While doing that it also disables its own line editing, setting the terminal in cooked mode and enabling the \r\n conversion. At that point, though, the input buffer already contains a\rb\r, which is what is read by cat and printed on the screen. If you then press Ctrl+D (^D), cat exits and a new prompt is printed, overwriting the (only) printed line (because it ends in a carriage return).

Note that:

  • If, in place of cat, you invoke a program that reads from standard input in a loop, equivalent to the script

    while IFS= read -r foo
    do
      # Accumulate the content of foo
    done
    

    it will read nothing until you hit Enter, because there is no \n character in the input buffer (unless the pasted data exceeds the terminal's input buffer size, see below).

  • After pasting, the text (b, in this case) you see on the line that follows the command is not the echo of the pasted content, but the output of cat. You can easily verify this, for instance, by pasting

    od -An -tc
    a
    b
    

    Pasting into a shell run as strace -f -e read,write bash may also help in understanding what is going on.

  • If the pasted text meant to be read from standard input is "enough" (on my system, more than 4095 characters; see "Canonical and noncanonical mode" in man 3 termios), what written above only applies to the first 4095 byte-sized read — what the terminal puts in its input buffer when in raw mode, at most. Subsequent reads happen in cooked mode, \r\n conversion takes place and that part of the pasted text is echoed to the command line (and likely intermingles with the command's output).

In principle, this behavior can be changed by disabling Bash's line editing:

$ set +o emacs
$ set +o vi
$ cat         # Pasting the above snippet here
a
b
a
b
$             # ^D makes cat exit and brings the prompt back

Note, though, that lengthy input will still intermingle with the output because the terminal has no way to tell the shell to wait until the paste operation is over before starting its processing, unless "bracketed paste" is enabled (see below).


zsh seems to be a bit simpler. The only pitfall: "bracketed paste" is probably enabled by default. It allows you to paste multi-line commands to the command line, executing them as if they were a script while ensuring that special characters (e.g. tabs, newlines) are properly handled. This also means that, when given

cat
a
b

zsh reads the three lines and executes cat, which waits for new input from the terminal; when cat exits, the shell tries running a (which fails, unless such a command exists on your system), then b.

Disabling bracketed paste allows for the behavior you seem to be looking for (but, compared to bash with no line editing, by default zsh does not echo the pasted content, except for the part that exceeds the first 4095 bytes (see the bulleted list above)):

% unset zle_bracketed_paste
% cat         # Pasting the above snippet here
a             # cat prints these two lines
b
%             # ^D makes cat exit and brings the prompt back

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.