Trailing spaces when copying from console

Question

It's this annoying behavior I've been experiencing here and there occasionally: when you select text with mouse in console (that is, copy it), paste it, and realize you've got extra spaces at the end of each line. That is,

line 1                                                                          
line 2                                                                          

instead of

line 1
line 2

So, not just one space at the end of each line.

I couldn't reliably reproduce the issue, and couldn't find the answer. I believe with some software it manifests itself only after a while.

But I've just noticed, that when I open the same file in vim, first right from console, then from tmux, it works out well in the former case. And doesn't in the latter one. Considering TERM=xterm-256color in console, and TERM=screen-256color in tmux, my conjecture is that it has to do with terminal not doing it properly, or not enabling applications to do it properly. Quite a vague conjecture, I presume. So, the first question is, "What exactly is causing it?"

And the other one is, "How do I go about it?" The worst case is when the file is located remotely. I used to copy it locally and open it with gedit lately. Now I supposedly have an option to open it in new console (since I'm mostly working in tmux sessions), and copy from there. Could this be done any simpler?

When I run vim from tmux with TERM=xterm-256color vim, it behaves strangely. Like not drawing background where there is no text. And it doesn't seem okay to me to change TERM variable (making software think it's dealing with other terminal).

When editing local file, I usually do :!gedit %.

Answer 1

You'll have spaces at the end of the line when selecting and copying from the terminal if the application displayed spaces at that spot. Applications may display spaces in order to erase what was there before. Terminals have commands to delete a whole line or delete characters to the right of the cursor; applications choose between that and displaying spaces based on what they consider most efficient. For example, if you type some stuff at a prompt, then press Backspace, the application (e.g. the shell) is likely to overwrite the last character with a space.

If you have an X11 connection, you can use xsel or xclip to copy a file to the local clipboard.

Experimentally, Vim seems to go through pains not to display lines ending with spaces (even when the buffer contains a line that ends with spaces). So copying from that is an option if you don't have an X11 connection.

An alternative would be to post-process after copying:

xsel | sed 's/  *$//' | xsel

Answer 2

That's actually several questions, and none of the answers do more than skim over the interesting parts:

• when you select text with mouse in console (that is, copy it), paste it, and realize you've got extra spaces at the end of each line.
• When I run vim from tmux with TERM=xterm-256color vim, it behaves strangely.

Most terminals (like xterm) store the data which you can select on the screen. There's no hidden part behind it which tells the terminal that the application meant to fill in a background.

Applications update the screen by moving the cursor around, writing text (which may include actual spaces — or tabs which the terminal renders as spaces) and erasing parts of the screen.

Erasure is a special problem (for select/paste), since many terminals (such as xterm) will fill the erased area of the screen with the current color (the back color erase (bce) feature in the terminal description). But at the same time, erased areas no longer have characters stored in those positions of the screen. Terminals which use the on-screen representation will let you select everything except for the erased area. (As a special case, your terminal could have erased regions surrounded by text, which it will pretend are spaces for selection).

All of that's a nuisance, and long ago xterm provided a feature for ignoring the trailing spaces. Most other terminals don't provide that. And since it's an option, it may not be turned on if you happened to run xterm. Most terminal developers that copy xterm's features don't copy the options.

Going to tmux: it doesn't support the bce feature. Its developers decided not to. So normal applications running in tmux will produce trailing blanks. When you override the terminal description, it confuses tmux, since it doesn't know that vim is really assuming that the background will be cleared using the current color. You just get the erasure. No color.

On the other hand, GNU screen's developers decided a while back to support the feature. It's optional...

Further reading:

• Support background color erase (bce) [was: Vim copy & paste trailing space issue] #109 (tmux bug report)
• My terminal shows some uncolored spaces (ncurses FAQ)
• The Trouble With Terminals (useful discussion on the topic)
• highlightSelection (xterm manual)
• trimSelection (xterm manual)

  If you set highlightSelection, you can see the text which is selected, including any trailing spaces. Clearing the screen (or a line) resets it to a state containing no spaces. Some lines may contain trailing spaces when an application writes them to the screen. However, you may not wish to paste lines with trailing spaces. If this resource is true, xterm will trim trailing spaces from text which is selected. It does not affect spaces which result in a wrapped line, nor will it trim the trailing newline from your selection. The default is "false".

• Patch #105 - 1999/6/5 - XFree86 3.9Pp (xterm changelog)

  implement new resource trimSelection, which allows xterm to trim trailing blanks from selected lines. This does not affect the highlighting.

• Patch #27 - 1996/8/21 - XFree86 3.1.2Ek (xterm changelog referring to highlightSelection)

  This patch fixes one of my long-term gripes: xterm's selection doesn't clearly show what's being selected (as per David's request, it's controlled by a resource, which defaults to the older behavior).

Answer 3

Copy & pasting from a terminal screen is never going to be fully reliable because it's dealing with screen output instead of original source material. If some applications echo text to the terminal in an unusual fashion and it results in the terminal not being able to guess what the original text was, there's probably not much you or the terminal can do about it.

A lot of information about the original text is potentially lost when it is rendered to a terminal: for example whether an area of white-space was produced by a tab or by a series of spaces, or whether two rows of text were originally one long line that got wrapped or two separate lines.

The terminal tries to do its best to let you copy & paste the original text that got echoed to the terminal but it cannot always know.

As an experiment, try this:

• Use less to view a file that contains very long lines that wrap across multiple terminal lines.
• Triple-click one of those lines (for select-whole-line). less selected the entire logical line that spans multiple physical lines and if you pasted it elsewhere it would be preserved as one long line.
• Press j some number of lines so that part of that long line disappears off the top of the screen.
• Press k one or more times to scroll the whole logical line back into view.
• Triple-click the logical line again. This time only one physical line is selected. That is because less repainted the screen physical line by physical line and the terminal no longer has any way of knowing that the physical lines were connected together.
• Now if you were to drag-select the entire logical line manually and paste it elsewhere, you would find that it has embedded newlines.

YMMV on that experiment because your terminal (or version of less) might be more or less clever than mine.

Generally, the dumber the software that produced the output, the better your chances are of being able to copy & paste exactly the original material. cat, for one, is pretty much as dumb as it gets.^{(You understand of course that "dumb" is a compliment!)}

If you get an extra space character at the end of a line under tmux, it's probably because tmux is actually echoing that character. Remember that tmux does its own terminal emulation and then re-emits new terminal sequences to render to the underlying terminal. Perhaps it is echoing that space character because it believes there might be a need in some cases to overwrite some other character it thinks might be there. Whatever the reason, the terminal probably has no way of knowing that space character is not really part of the original content.

Answer 4

In a lot of modern Terminal applications there are settings like "Trim trailing spaces" in mouse/copy options. Those are present for Konsole (KDE) & Yakuake.

Answer 5

Using Fedora 17 Linux and the konsole terminal window, I open vim with the syntastic vim add-on, I was able to reproduce this error consistently. I put these three python lines in vim:

a = "generic assignment"
b = "cursor is on this line"
c = "generic assignment"

Then I visually select those lines and paste it here:

a = "generic assignment"
b = "cursor is on this line"                                                                                     
c = "generic assignment"

Notice how the second line has tons of spaces copied over. That sure gets annoying.

Lines 1 and 3 are copied as expected, but the second line copies spaces until the rightmost spot of the terminal window.

Work-around Solution:

Move the highlighted line (the vim cursor) away from the lines you want to copy before selecting it with the mouse. Then the extra spaces don't appear on that one line.

I'm guessing it is the syntax and color highlighting add-ons that are causing these issues.

Answer 6

I also use xterm-256color and always did not like the trailing spaces when copying between terminals, for example copying a whole file between two remote servers. What I find easiest is just to copy the whole file and then in the destination run:

:%s/\s\+$//g

This will remove all trailing whitespaces from the whole file. It is fast and easy.

Answer 7

xterm-256color seems to work properly now (full colours and no background drawing issues) and has BCE (back color erase) functionality. You can enable it in tmux by putting this in your ~/.tmux.conf:

set -g default-terminal "xterm-256color"