How to erase unicode symbols in terminal?

Question

Some symbols take up two character cells. Consider this script:

#!/usr/bin/env bash
echo '银^Htest'
echo 'а^Htest'

Output:

 test
test

How do I know their code points of fullwidth symbols? Some sort of regex for that? How do I count how many character cells a string takes? How do I erase everything that was outputted?

I'm using xterm if that matters.

UPD To give you the bigger picture, I'm trying to display progress, outputting some info, erasing it, outputting again... For that, I was moving cursor to the beginning of the line (\r), erasing the line with spaces (tput cols), and moving the cursor again (\r). But it turned out, that output might span several lines. So I decided to count characters, move back (^H), erase (output space as many times as the length of the string is), and move back again (^H).

Answer 1

That is actually four questions:

1. How do I know their code points?
2. Some sort of regex?
3. How do I count how many character cells a string takes?
4. How do I erase everything that was outputted?

OP mentions xterm, but only the last two are possibly specific to xterm.

For (1) and (2), the echo command is not much help. You are better off using printf, which recognizes backslash escapes. In some implementations (e.g., GNU coreutils), that includes Unicode constants, e.g.,

printf '\u94f6\btest'

although for regular expressions, you are yet again better off using a scripting language such as Perl (which can handle UTF-8).

Further reading:

• How do you echo a 4-digit Unicode character in Bash?
• The printf command
• What's the best way to embed a Unicode character in a POSIX shell script?
• Bash equivalent to Python's string literal for utf string conversion

Questions (3) and (4) are more interesting. First, the script cannot really tell how many character cells a string takes in advance, but can only measure it after the fact. That is because the width is based on a combination of behavior of the terminal and the kernel.

• xterm uses wcwidth to decide how wide the character should be, with some issues about "wide" (double-width) fonts, and implementations of wcwidth reflecting developer's biases regarding the ambiguous-width Unicode values. xterm can be configured (at runtime) to use its copy of Markus Kuhn's implementation of wcwidth; you are cautioned that it may not be complete nor match the actual system locale information.
• when xterm is told to erase part of a double-width character (as in the given example), it replaces the other part with a space. Most other terminals imitating xterm do this (though in a quick check, I noticed one simply moving the cursor, causing the wide character and ASCII text to overlap). If you knew that the value was double-width, you could simply adjust your notion of where the cursor was.
• the Linux kernel does not know anything about wcwidth. Linux-based systems since 2004 have a feature in stty called iutf8:

which tells the kernel that input is encoded in UTF-8, for proper editing support in canonical input mode

• the Linux kernel feature is useful for editing input, because it helps the terminal driver do something reasonable when a backspace deletes the previous character. However, there is no comparable feature for output.

You can, as suggested, use the cursor-position report (an escape sequence) to find the position of the cursor at different points. But if you are going to use that to decide how to clear the line, it might seem more direct to just move to the position before printing the Unicode value, and start clearing from that point.

Alternatively, you could tell the terminal to save the cursor position before printing the wide character, and restore it (moving back). That might seem cleaner and more predictable. After restoring the cursor position, you could clear the line. All three could be done using escape sequences — or tput:

tput sc
printf '\u94f6'
tput rc
tput el
printf 'test'

Other than as a demonstration, this does have the drawback that the terminal would have only one saved-position for the cursor, and that to ensure clearing a single/double-width character, it is clearing an entire line. But the cursor would end up in the "right" place.

• Get the display width of a string of characters
• terminfo - terminal capability data base (for sc, el and rc)