2

Some symbols take up two character cells. Consider this script:

#!/usr/bin/env bash
echo '银^Htest'
echo 'а^Htest'

Output:

 test
test

How do I know their code points of fullwidth symbols? Some sort of regex for that? How do I count how many character cells a string takes? How do I erase everything that was outputted?

I'm using xterm if that matters.

UPD To give you the bigger picture, I'm trying to display progress, outputting some info, erasing it, outputting again... For that, I was moving cursor to the beginning of the line (\r), erasing the line with spaces (tput cols), and moving the cursor again (\r). But it turned out, that output might span several lines. So I decided to count characters, move back (^H), erase (output space as many times as the length of the string is), and move back again (^H).

2

That is actually four questions:

  1. How do I know their code points?
  2. Some sort of regex?
  3. How do I count how many character cells a string takes?
  4. How do I erase everything that was outputted?

OP mentions xterm, but only the last two are possibly specific to xterm.

For (1) and (2), the echo command is not much help. You are better off using printf, which recognizes backslash escapes. In some implementations (e.g., GNU coreutils), that includes Unicode constants, e.g.,

printf '\u94f6\btest'

although for regular expressions, you are yet again better off using a scripting language such as Perl (which can handle UTF-8).

Further reading:

Questions (3) and (4) are more interesting. First, the script cannot really tell how many character cells a string takes in advance, but can only measure it after the fact. That is because the width is based on a combination of behavior of the terminal and the kernel.

  • xterm uses wcwidth to decide how wide the character should be, with some issues about "wide" (double-width) fonts, and implementations of wcwidth reflecting developer's biases regarding the ambiguous-width Unicode values. xterm can be configured (at runtime) to use its copy of Markus Kuhn's implementation of wcwidth; you are cautioned that it may not be complete nor match the actual system locale information.
  • when xterm is told to erase part of a double-width character (as in the given example), it replaces the other part with a space. Most other terminals imitating xterm do this (though in a quick check, I noticed one simply moving the cursor, causing the wide character and ASCII text to overlap). If you knew that the value was double-width, you could simply adjust your notion of where the cursor was.
  • the Linux kernel does not know anything about wcwidth. Linux-based systems since 2004 have a feature in stty called iutf8:

which tells the kernel that input is encoded in UTF-8, for proper editing support in canonical input mode

  • the Linux kernel feature is useful for editing input, because it helps the terminal driver do something reasonable when a backspace deletes the previous character. However, there is no comparable feature for output.

You can, as suggested, use the cursor-position report (an escape sequence) to find the position of the cursor at different points. But if you are going to use that to decide how to clear the line, it might seem more direct to just move to the position before printing the Unicode value, and start clearing from that point.

Alternatively, you could tell the terminal to save the cursor position before printing the wide character, and restore it (moving back). That might seem cleaner and more predictable. After restoring the cursor position, you could clear the line. All three could be done using escape sequences — or tput:

tput sc
printf '\u94f6'
tput rc
tput el
printf 'test'

Other than as a demonstration, this does have the drawback that the terminal would have only one saved-position for the cursor, and that to ensure clearing a single/double-width character, it is clearing an entire line. But the cursor would end up in the "right" place.

  • The first two questions are basically, "What code points do fullwidth characters have?". With this being the best answer I could find. Then, as you could have already seen (in my question), the output might span several lines, so after all I most likely need to ask for cursor position to be able to find out which lines I need to erase. And then comes the scrolling issue. While outputting, the text might scroll. So, stored position might be no longer pointing to the beginning of the output. Any clues? – x-yuri Jan 29 '16 at 7:51
  • In a shell script, you're probably going to have problems. You also probably should ask more focused questions, since there are a lot of possible answers. – Thomas Dickey Jan 29 '16 at 10:12
  • How about ruby or php? Or you can probably use language of your choosing. Can you help me make it more focused? I don't really understand what I need to provide. – x-yuri Jan 29 '16 at 12:08
  • 1
    To get the width of a one-line string according to wcwidth, you can use the shell command wc -L. – egmont Jan 29 '16 at 14:35
  • 1
    Also let's not forget that an additional cell will be left empty at the end of the line if a wide character is printed there (it is wrapped to the next line). This makes tracking the number of required backspaces quite cumbersome. (Let alone that backspacing over a line boundary is another tricky story.) – egmont Jan 29 '16 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.