Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

The unit separator ASCII character (ASCII 31, octal 37), is visible in Vim as a ^_. But if I print the same file to the terminal, the character is invisible. This causes the fields on a line to get stuck together:

# In Vim and less:

first field^_second field^_last field

# cat the same file to terminal:
cat delim.txt
first fieldsecond fieldlast field

# print 2nd field with awk 
cat delim.txt | awk 'BEGIN {FS = "\037"} {print $2}'
second field

I suppose I can make the unit separator visible with cat -v:

cat -v delim.txt
first field^_second field^_last field

But this is rather cumbersome. Why doesn't the unit separator have a visible representation when printed to stdout in the Bash shell? I can't even copy and paste the shell output correctly; the unit separator gets lost in the process.

share|improve this question
    
Not all characters are printable, unit separator is one of these. Some editors will display it in some way to make editing possible. You need to translate it into a sequence of printable characters, and maybe a different font/colour, to reduce ambiguity. –  richard May 5 at 20:58
1  
ASCII codes under 31, and 127, are intended to cause a terminal or device to do something (hence why they are called control codes), or stand for something in a protocol (like EOT or SOH), as opposed to displaying something. It hearkens back when terminals were typewriter-like devices and things like telling a teletype to carriage-return were physically needed. Editors may choose to render them using "^" notation since you are editing something and don't want to terminal to actually do what the control codes are asking. –  ultrasawblade May 6 at 2:20

3 Answers 3

up vote 11 down vote accepted

The unit separator (US) character, also known as IS1, is in the cntrl character class and is not in the print character class. It is a control character that is intended for organizing text into groups, for programs that are designed to make use of that information. In general, non-printable characters are probably going to be interpreted and rendered differently in different programs or environments.

The reason you are seeing it represented as ^_ in Vim is because Vim is an interactive editor. It can freely render non-printable characters however it wants, as long as the correct binary character is written to disk.

You cannot get the same behavior in the shell because Unix shell programs are written to operate on and pass plain text to each other. When you cat a file, the text that is written to the terminal must be what is actually in the file.

So that leaves it to the terminal device to interpret the character. And it turns out that some terminal emulators do render the US character differently from others. In gnome-terminal (or any vte-based terminal), the character will be rendered as a box containing the hex code 001F. In xterm or rxvt, the character is indeed invisible.

share|improve this answer

A little bit at the margin of the other (very good) answers, if you want to alter only the control character ^_ when displaying the file content, you might want to transliterate it using the tr utility (and a little bit of bash-compatible syntax):

# Replace the control character US (^_) by *one* other character
$ cat my.file | tr $'\c_' ':'

If you need to replace that control character by its "expanded" form, you will need sed instead:

# Replace the control character US (^_) by any string
cat /tmp/f | sed s/$'\c_'/^_/g

Please note the syntax $'\cX': this syntax inform your (bash-compatible shell) to replace the corresponding control character. See wikipedia for a list of control characters alias using the "caret notation". If you don't like that syntax, you might prefer using the octal $'\037' or hexadecimal $'\x31' notation instead.

share|improve this answer

The unit separator is in the ASCII range of Control Characters, and therefore does not (or should not usually) have a visual representation.

Vim and some other editors display them, so you can edit them. As you noticed, cat -v displays it too. The man page shows, that -v is the short form of --show-nonprinting, which causes it to replace the non-printing characters with a printable representation, which is not the original content of the file and might therefore cause trouble, if the output is actually to another program.

The representation you see already hints it's a control character: a character prepended with a ^ is a common notation for Ctrl + the character, which is the key combination that produces this character in a terminal. Ctrl+_ will let you input the unit separator in vim, for example. But another editor or some GUI viewer might display the hex code, a placeholder or something completely different.

As your terminal does not print the control characters, it is also not copied when selecting the text (the whitespace characters like newline and tab are an exception here, which are control characters too). Another example of control characters in the terminal that are usually ignored when copying are color codes, which are an ESC character followed by the code for coloring the text.

So to show the characters on your terminal, there is no other way than to use a program that replaces the unit separator with some printable character.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.