7

It's this annoying behavior I've been experiencing here and there occasionally: when you select text with mouse in console (that is, copy it), paste it, and realize you've got extra spaces at the end of each line. That is,

line 1                                                                          
line 2                                                                          

instead of

line 1
line 2

So, not just one space at the end of each line.

I couldn't reliably reproduce the issue, and couldn't find the answer. I believe with some software it manifests itself only after a while.

But I've just noticed, that when I open the same file in vim, first right from console, then from tmux, it works out well in the former case. And doesn't in the latter one. Considering TERM=xterm-256color in console, and TERM=screen-256color in tmux, my conjecture is that it has to do with terminal not doing it properly, or not enabling applications to do it properly. Quite a vague conjecture, I presume. So, the first question is, "What exactly is causing it?"

And the other one is, "How do I go about it?" The worst case is when the file is located remotely. I used to copy it locally and open it with gedit lately. Now I supposedly have an option to open it in new console (since I'm mostly working in tmux sessions), and copy from there. Could this be done any simpler?

When I run vim from tmux with TERM=xterm-256color vim, it behaves strangely. Like not drawing background where there is no text. And it doesn't seem okay to me to change TERM variable (making software think it's dealing with other terminal).

When editing local file, I usually do :!gedit %.

3

You'll have spaces at the end of the line when selecting and copying from the terminal if the application displayed spaces at that spot. Applications may display spaces in order to erase what was there before. Terminals have commands to delete a whole line or delete characters to the right of the cursor; applications choose between that and displaying spaces based on what they consider most efficient. For example, if you type some stuff at a prompt, then press Backspace, the application (e.g. the shell) is likely to overwrite the last character with a space.

If you have an X11 connection, you can use xsel or xclip to copy a file to the local clipboard.

Experimentally, Vim seems to go through pains not to display lines ending with spaces (even when the buffer contains a line that ends with spaces). So copying from that is an option if you don't have an X11 connection.

An alternative would be to post-process after copying:

xsel | sed 's/  *$//' | xsel
  • So copying from that is an option if you don't have an X11 connection. Not with screen-like terminal emulators, AFAICT (TERM=screen*). Speaking of xsel, good point, I'm now thinking of creating a vim command, which outputs shell command to be issued locally, like :XSelCommand % -> ssh server cat path/to/file | xsel. That seems better, than mounting remote filesystems, or copying files locally. – x-yuri Jul 26 '15 at 19:20
  • So, with a command like this: command! XSelCommand echo 'ssh ' . system('hostname')[:-2] . ' ' . shellescape('sh -c ''cat "$1"'' -- ' . shellescape(expand('%'))) . ' | xsel', or like this one: command! -range=% RXSelCommand echo 'ssh ' . system('hostname')[:-2] . ' ' . shellescape('sh -c ''cat "$1" | sed -n "<line1>,<line2>p"'' -- ' . shellescape(expand('%'))) . ' | xsel', one can copy files' contents, not what was output. The easier option, however, would be rather to open the file in local vim over ssh and do :%!xsel. Or do :r!ssh HOST CMD, and again :%!xsel. – x-yuri Jul 29 '15 at 0:36
  • I've managed to run into two issues with xsel so far: 1) :%!xsel doesn't work in vim, but :%!xsel -i does work, supposedly because vim redirects stdout as well as stdin; one more option is :w !xsel, 2) copying files over 4000 bytes doesn't work when pasting to chrome's textarea for example, but works for gedit; the tab hangs for 10 or so seconds, but nothing is pasted after that; do you know by any chance what kind of magic number that is? – x-yuri Sep 16 '15 at 17:36
  • ...And nothing of the sort happens when copying from gedit to chrome with middle mouse button. – x-yuri Sep 16 '15 at 17:43
  • @x-yuri I don't know why there would be a size limit. Try changing xsel/middle-mouse-button to xsel -b/Ctrl+V, that uses a slightly different selection mechanism which may be differently buggy. – Gilles Sep 16 '15 at 18:00
3

That's actually several questions, and none of the answers do more than skim over the interesting parts:

  • when you select text with mouse in console (that is, copy it), paste it, and realize you've got extra spaces at the end of each line.
  • When I run vim from tmux with TERM=xterm-256color vim, it behaves strangely.

Most terminals (like xterm) store the data which you can select on the screen. There's no hidden part behind it which tells the terminal that the application meant to fill in a background.

Applications update the screen by moving the cursor around, writing text (which may include actual spaces — or tabs which the terminal renders as spaces) and erasing parts of the screen.

Erasure is a special problem (for select/paste), since many terminals (such as xterm) will fill the erased area of the screen with the current color (the back color erase (bce) feature in the terminal description). But at the same time, erased areas no longer have characters stored in those positions of the screen. Terminals which use the on-screen representation will let you select everything except for the erased area. (As a special case, your terminal could have erased regions surrounded by text, which it will pretend are spaces for selection).

All of that's a nuisance, and long ago xterm provided a feature for ignoring the trailing spaces. Most other terminals don't provide that. And since it's an option, it may not be turned on if you happened to run xterm. Most terminal developers that copy xterm's features don't copy the options.

Going to tmux: it doesn't support the bce feature. Its developers decided not to. So normal applications running in tmux will produce trailing blanks. When you override the terminal description, it confuses tmux, since it doesn't know that vim is really assuming that the background will be cleared using the current color. You just get the erasure. No color.

On the other hand, GNU screen's developers decided a while back to support the feature. It's optional...

Further reading:

  • Support background color erase (bce) [was: Vim copy & paste trailing space issue] #109 (tmux bug report)
  • My terminal shows some uncolored spaces (ncurses FAQ)
  • The Trouble With Terminals (useful discussion on the topic)
  • highlightSelection (xterm manual)
  • trimSelection (xterm manual)

    If you set highlightSelection, you can see the text which is selected, including any trailing spaces. Clearing the screen (or a line) resets it to a state containing no spaces. Some lines may contain trailing spaces when an application writes them to the screen. However, you may not wish to paste lines with trailing spaces. If this resource is true, xterm will trim trailing spaces from text which is selected. It does not affect spaces which result in a wrapped line, nor will it trim the trailing newline from your selection. The default is "false".

  • Patch #105 - 1999/6/5 - XFree86 3.9Pp (xterm changelog)

    implement new resource trimSelection, which allows xterm to trim trailing blanks from selected lines. This does not affect the highlighting.

  • Patch #27 - 1996/8/21 - XFree86 3.1.2Ek (xterm changelog referring to highlightSelection)

    This patch fixes one of my long-term gripes: xterm's selection doesn't clearly show what's being selected (as per David's request, it's controlled by a resource, which defaults to the older behavior).

  • The general idea seems to be clear, but I might not be understanding the particulars of some statements, or not sure if I do, like... Terminals which use the on-screen representation What are other types of terminals? Are they covered in your answer? As a special case, your terminal could have erased regions surrounded by text, which it will pretend are spaces for selection Here you probably mean "marked as erased". And what kind of regions are these? Spaces in the middle of the line? Empty lines surrounded by text? These are probably the least clear ones. – x-yuri Jan 2 '17 at 9:52
  • Other types: applications (such as curses) which do higher-levels "know" if they're clearing the whole screen or repainting a line, and have the data to reconstruct things in more than one way. "Marked as erased": for example xterm may show a blank for erased areas, but actually has no character. – Thomas Dickey Jan 2 '17 at 11:29
  • fwiw, "screen" and "tmux" are common examples of applications which can manipulate the terminal so that selection behaves differently. – Thomas Dickey Jan 2 '17 at 11:44
2

Copy & pasting from a terminal screen is never going to be fully reliable because it's dealing with screen output instead of original source material. If some applications echo text to the terminal in an unusual fashion and it results in the terminal not being able to guess what the original text was, there's probably not much you or the terminal can do about it.

A lot of information about the original text is potentially lost when it is rendered to a terminal: for example whether an area of whitespace was produced by a tab or by a series of spaces, or whether two rows of text were originally one long line that got wrapped or two separate lines.

The terminal tries to do its best to let you copy & paste the original text that got echoed to the terminal but it cannot always know.

As an experiment, try this:

  • Use less to view a file that contains very line lines that wrap across multiple terminal lines.
  • Triple-click one of those lines (for select-whole-line). less selected the entire logical line that spans multiple physical lines and if you pasted it elsewhere it would be preserved as one long line.
  • Press j some number of lines so that part of that long line disappears off the top of the screen.
  • Press k one or more times to scroll the whole logical line back into view.
  • Triple-click the logical line again. This time only one physical line is selected. That's because less repainted the screen physical line by physical line and the terminal no longer has any way of knowing that the physical lines were connected together.
  • Now if you were to drag-select the entire logical line manually and paste it elsewhere, you would find that it has embedded newlines.

YMMV on that experiment because your terminal (or version of less) might be more or less clever than mine.

Generally, the dumber the software that produced the output, the better your chances are of being able to copy & paste exactly the original material. cat, for one, is pretty much as dumb as it gets.(You understand of course that "dumb" is a compliment!)

If you get an extra space character at the end of a line under tmux, it's probably because tmux is actually echoing that character. Remember that tmux does its own terminal emulation and then re-emits new terminal sequences to render to the underlying terminal. Perhaps it is echoing that space character because it believes there might be a need in some cases to overwrite some other character it thinks might be there. Whatever the reason, the terminal probably has no way of knowing that space character is not really part of the original content.

  • You made me come up with a workaround: cat a file, and copy the output of cat. At least in my particular case it works. Thanks :) But if it's a long file... Well, I was thinking about mounting remote filesystem (sshfs), then I can edit a file with gedit for example as if it were a local one. – x-yuri Jul 25 '15 at 7:59
  • If data you want to copy is in the file, you can use xsel -bi < filename to safely copy all of it... Use recent version of xsel, though – Matija Nalis Sep 6 '18 at 16:16
2

In a lot of modern Terminal applications there are settings like "Trim trailing spaces" in mouse/copy options. Those are present for Konsole (KDE) & Yakuake.

1

Using Fedora 17 Linux and the konsole terminal window, I open vim with the syntastic vim add-on, I was able to reproduce this error consistently. I put these three python lines in vim:

a = "generic assignment"
b = "cursor is on this line"
c = "generic assignment"

Then I visually select those lines and paste it here:

a = "generic assignment"
b = "cursor is on this line"                                                                                     
c = "generic assignment"

Notice how the second line has tons of spaces copied over. That sure gets annoying.

Lines 1 and 3 are copied as expected, but the second line copies spaces until the rightmost spot of the terminal window.

Work-around Solution:

Move the highlighted line (the vim cursor) away from the lines you want to copy before selecting it with the mouse. Then the extra spaces don't appear on that one line.

I'm guessing it is the syntax and color highlighting add-ons that are causing these issues.

  • I got line 1 padded with spaces as well. – x-yuri Nov 22 '15 at 0:13
0

xterm-256color seems to work properly now (full colours and no background drawing issues) and has BCE (back color erase) functionality. You can enable it in tmux by putting this in your ~/.tmux.conf:

set -g default-terminal "xterm-256color"

0

I also use xterm-256color and always did not like the trailing spaces when copying between terminals, for example copying a whole file between two remote servers. What I find easiest is just to copy the whole file and then in the destination run:

:%s/\s\+$//g

This will remove all trailing whitespaces from the whole file. It is fast and easy.

  • Occasionally I get less lines then in the original (newlines doesn't always propagate). You might want to check this after doing the replacements. – x-yuri Feb 25 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.