So I have a lot of data WITHOUT NEW LINES on the clipboard (it's a large SVG file on one line). I went

$ cat >file.svg

then tried to paste (in Gnome Terminal), but only the first 4kB characters were accepted.

I assume this is a readline feature/limitation.

Is there a way to read from STDIN that would avoid this problem?

EDIT

Test case: Create a demo file. This one will have ~4k "=" symbols followed by "foo bar".

{ printf '=%.0s' {1..4095} ; echo "foo bar" ; } > test.in

Copy that into your clipboard

xclip test.in

(if you want to middle-click to insert) or

xclip -selection clipboard test.in

(if you want to use Ctrl-Shift-Insert to past it in)

Then cat >test.out, paste (whichever way). Press Ctrl-D to end the stream. cat test.out - do you see "foo bar"?

On my set-up (Ubuntu 12.04, Gnome Terminal, zsh) when I paste I only see the = and I don't see foo bar. Same when I inspect test.out.

  • Are you sure your SVG file was entirely read into your clipboard? – lgeorget May 20 '14 at 11:31
  • What is your actual problem? How to store the content of the clipboard to a file? If so, there are other way than pasting in the terminal. – lgeorget May 20 '14 at 11:33
  • how much is N in your case? I've tried it with 2kB of xml data (inc LF) no problem. – fduff May 20 '14 at 11:34
  • 1
    @artfulrobot A foreground process interacts directly with the tty/pty. The shell is not involved. You can see this because you do not have readline features (editing/jumping commands, history, ...) in programs if they do not use readline or any other input library themselves. – jofel May 20 '14 at 11:55
  • 1
    This isn't a readline limitation — readline and bash aren't involved here. It's a limitation of the terminal interface. – Gilles May 20 '14 at 22:57
up vote 17 down vote accepted

If I understand the source correctly, under Linux, the maximum number of characters that can be read in one go on a terminal is determined by N_TTY_BUF_SIZE in the kernel source. The value is 4096.

This is a limitation of the terminal interface, specifically the canonical (“cooked”) mode which provides an extremely crude line editor (backspace, enter, Ctrl+D at the start of a line for end-of-file). It happens entirely outside the process that's reading.

You can switch the terminal to raw mode, which disables line processing. It also disables Ctrl+D and other niceties, putting an extra burden on your program.

This is an ancient Unix limitation that's never been fixed because there's little motivation. Humans don't enter such long lines. If you were feeding input from a program, you'd redirect your program's input from a file or a pipe.

For example, to use the content of the X clipboard, pipe from xsel or xclip. In your case:

xsel -b >file.svg
xclip -selection clipboard >file.svg

Remove -b or -selection clipboard to use the X selection (the one that is set by highlighting with the mouse) rather than the clipboard.

On OSX, use pbpaste to paste the clipboard content (and pbcopy to set it).

You can access the X clipboard over SSH if you activate X11 forwarding with ssh -X (which some servers may forbid). If you can only use ssh without X11 forwarding, you can use scp, sftp or sshfs to copy a file.

If pasting is the only solution because you can't forward the clipboard or you aren't pasting but e.g. faking typing into a virtual machine, an alternative approach is to encode the data into something that has newlines. Base64 is well-suited for this: it transforms arbitrary data into printable characters, and ignores whitespace when decoding. This approach has the additional advantage that it supports arbitrary data in the input, even control characters that the terminal would interpret when pasting. In your case, you can encode the content:

xsel -b | base64 | xsel -b

then decode it:

base64 -d
Paste
Ctrl+D

The limit you're running into is the maximum size of a line in canonical input mode, MAX_CANON.

In canonical input mode, the tty driver provides basic line editing services so the userspace program doesn't need to. It doesn't have nearly as many features as readline, but it recognizes a few configurable special characters like erase (usually Backspace or Delete) and kill (usually Ctrl-U).

Most importantly for your question, canonical mode buffers input until the end-of-line character is seen. Because the buffer is in the tty driver, in kernel memory, it's not very large.

You can turn off canonical mode with stty cbreak or stty -icanon, and then do your paste. This has the significant disadvantage that you will not be able to send an EOF with Ctrl-D. That's another one of the things that canonical mode is responsible for. You will still be able to terminate the cat with Ctrl-C because the signal-generating characters are controlled by a separate flag (stty raw or stty -isig).

The mystery to me is why, since you've already demonstrated that you know about xclip, you don't just use xclip -o > file instead of the cat

  • 1
    The mystery can be easily solved: It seems that artfulrobot wants to quickly fill a file on a remote hosts with data from the clipboard. In the remote shell, there is normally no direct access to the local clipboard via xclip. – jofel May 21 '14 at 9:00
  • 2
    Ah, good old upload-by-pasting. If I had to do one of those and it wasn't plain text, I'd uuencode it instead of trying to convince the tty driver to pass it through. Plain text with enormous lines could be handled that way too. – Wumpus Q. Wumbley May 21 '14 at 12:11

If you do:

stty eol =

And then run the demo suggested in your EDIT, you will see foo bar in the printout of test.out. The terminal's line discipline will flush its output to its reader as it reads each special eol char in your input.

A Linux canonical-mode terminal - as can be configured with stty icanon or probably just stty sane - handles the following special input characters...

  • eof
    • default: ^D
    • Terminates an input line and flushes output to the reader. Because it is removed from input, if it is input as the only character on a line, it is passed as a null read - or end of file - to the reader.
  • eol
    • default: unsassigned
    • Also terminates an input line, but is not removed from input.
  • kill
    • default: ^U
    • Erases all buffered input.
  • erase
    • default: ^H (or possibly @ or ^? on some systems)
    • Erases the last buffered input character.

When iexten is also set - like stty icanon iexten or, again, probably just stty sane, a canonical Linux terminal will also handle...

  • eol2
    • default: unassigned
    • Also also terminates an input line, and is also not removed from input.
  • werase
    • default: ^W
    • Erases the last buffered input word.
  • rprnt
    • default: ^R
    • Reprints all buffered input.
  • lnext
    • default: ^V
    • Removes any special significance as far as the line-discipline is concerned for the immediately following input character.

These characters are handled by removing them from the input stream - excepting eol and eol2, that is - and performing the associated special function before passing the processed stream to the reader - which is usually your shell, but could be whatever the foreground process group is.

Other special input characters which are similarly handled but can be configured independently of any icanon setting include the isig set - set like stty isig and probably also included in a sane configuration:

  • quit
    • default: ^\
    • Flushes all buffered input (if noflsh is not set) and sends SIGQUIT to the foreground process-group - likely generating a core-dump.
  • susp
    • default: ^Z
    • Flushes all buffered input (if noflsh is not set) and sends SIGTSTP to the foreground process-group. The suspended process-group can likely be resumed with either of kill -CONT "$!" or just fg in a (set -m) job-controlled shell.
  • intr
    • default: ^C
    • Flushes all buffered input (if noflsh is not set) and sends SIGINT to the foreground process-group.

And the ixon set - configured like stty ixon and also usually included in a sane config:

  • stop
    • default: ^S
    • Stops all output to the reader until either start is read in input or - when ixany is also set - at least one more character is read.
  • start
    • default: ^Q
    • Restarts output if it has previously been stopped with stop.
  • Both of stop and start are removed from input when processed, but if output is restarted due to any character in input when ixany is set then that character is not removed.

Special characters handled on other non-Linux systems might include...

  • flush
    • default: ^O
    • Toggles the discarding and flushing of buffered input and is removed from input.
  • dsusp
    • default: unassigned
    • Flushes all buffered input only when the reader reads the assigned special input character then sends SIGTSTP.

And possibly...

  • swtch
    • default ^@ (meaning \0 or NUL)
    • Switches foreground shell-layers. For use with the shl shell-layers application on some systems.
    • An implementation of shl which multiplexes ptys and is therefore compatible with job-control rather than the original implementation's swtch dependent behavior can be freely had in the heirloom-toolchest tool suite.

For a clearer picture of how and why (and perhaps why not) these input functions are handled consult man 3 termios.

All of the above functions can be assigned (or reassigned) - when applicable - like sttyfunction assigned-key. To disable any single function do sttyfunction^-. Alternatively, as various attempts with assignments for any of the aforementioned line-editing functions with all of GNU, AST, or heirloom's stty implementations seem to indicate, you can also sttyfunction^@ as NUL assignment for any function seems to equate to setting it to unassigned on my linux system.

Probably you do see an echo of these characters when you type them (as can likely be configured w/ [-]ctlecho), but this is only a marker to show you where you did - the program receiving your input has no notion that you typed them (excepting eol[2], that is) and receives only a copy of your input to which the line discipline has applied their effects.

A consequence of the terminal's handling of the various line-editing functions is that it must needs buffer the input to some extent in order to act upon the functions you indicate to it that it should - and so there cannot be a limitless supply of input which you might at any time kill. The line buffer is more precisely the kill buffer.

If you set the eol or eol2 characters to some delimiter which occurs in input - even if neither is a newline or a return character, for example - then you will only be able to kill up to the point that it last occurred and your kill buffer will extend as far as it can until the next of these - or a newline (or return if icrnl is set and igncr is not) - occurs in input.

cat will accept any number of characters, as you could witness by doing for example cat /dev/random > test.bin (don't do it unless you know how to stop it :). I tried copying and pasting a big file into cat > test.txt. All of the lines ended up in the file whether I cancelled with Ctrl-c or Ctrl-d, but in the former case not all the lines were printed to the terminal. This I believe is because cat buffers its printing, waiting for a complete buffer of text or direct input from the terminal before each printing.

On my system, I think the buffer size is 4096 (2^12) bytes: Create a file of 4095 bytes using (printf '1234567890%.0s' {1..409} && printf 12345) > test.in, load that into the copy buffer using xclip test.in, start cat > test.out, paste using Shift-Insert, and terminate the stream by pressing Ctrl-d. Now add a byte using printf '6' >> test.in, and the stream is printed twice: Once in the cat output (all 4096 bytes), and the last 4095 bytes again on the shell after terminating.

  • +1 In my case, it also depended on the clipboard used. If I used the selection buffer (middle click paste) I only saw the first 4542 lines of my test data (but all of them ended up in the created file) but using the X clipboard (Ctrl+C/Ctrl+V) I saw all of it. In both cases all data were printed into the resulting file but in the former only partial data were shown in the terminal. – terdon May 20 '14 at 12:03
  • I don't get same behaviour. See edited question – artfulrobot May 20 '14 at 12:28

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.