Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a list of Unicode codepoints, but I don't know of a "simple" way to convert these hex values into the actual characters they represent...

I've heard that zsh has echo -e '\u0965', but I use bash

Is there something as simple as the zsh method, for bash?

share|improve this question
1  
add comment

4 Answers

up vote 5 down vote accepted

You can use bash's echo or /bin/echo from GNU coreutils in combination with iconv:

echo -ne '\x09\x65' | iconv -f utf-16be

By default iconv converts to your locales encoding. Perhaps more portable than relying on a specific shell or echo command is Perl. Most any UNIX system I am aware of while have Perl available and it even have several Windows ports.

perl -C -e 'print chr 0x0965'

Most of the time when I need to do this, I'm in an editor like Vim/GVim which has built-in support. While in insert mode, hit Ctrl-V followed by u, then type four hex characters. If you want a character beyond U+FFFF, use a capital U and type 8 hex characters. Vim also supports custom easy to make keymaps. It converts a series of characters to another symbol. For example, I have a keymap I developed called www, it converts TM to ™, (C) to ©, (R) to ®, and so on. I also have a keymap for Klingon for when that becomes necessary. I'm sure Emacs has something similar. If you are in a GTK+ app which includes GVim and GNOME Terminal, you can try Control-Shift-u followed by 4 hex characters to create a Unicode character. I'm sure KDE/Qt has something similar.

share|improve this answer
    
Thanks... the perl one in nice and terse, but it has me a bit puzzled as to how it knows to treat the value as UTF-16BE.. I guess that's what the "chr" means... –  Peter.O Apr 29 '11 at 9:08
    
@fred that's a good point. The Perl example is locale sensitive. The -C enables full Unicode processing, but the example works because my locale uses a Unicode example. If I set LANG to C, I get a warning about a wide character in print, but it still prints. If I print chr 0xa2 in a UTF-8 locale I get a cents sign ¢, but if I use LANG=C, I get � because it prints out the byte 0xa2 which is invalid in UTF-8. The Vim/GVim example is semi-sensitive to the locale. More correctly, to the file encoding. If you started Vim in a non-UTF-8 locale, you will need to :set encoding=utf-8 –  penguin359 Apr 29 '11 at 10:41
    
@fred I should point out, Perl treat the value of chr as a Unicode codepoint if Perl is started in a Unicode locale like UTF-8. A codepoint is the unique number that represents a character and is not tied to any one encoding such as UTF-16BE or UTF-8. It converts it to the correct encoding when it prints it out. For example, the Cuneiform Sign A is codepoint U+012000. I can use chr 0x12000 in Perl (assuming Unicode is active) to represent it. In UTF-16BE, this is 0xd8, 0x08, 0xdc, and 0x00. You character is U+0965 which just happens to be the bytes 0x09 followed by 0x65 in UTF-16BE. –  penguin359 Apr 29 '11 at 17:25
    
@penguin359.. Thanks, one day (hopefully) I'll have a good look at perl.. It seems unfathomably cryptic, but then so did sed and regex, initially, and now it is quite easy... maybe it's a bit like vim; a steep learning curve, then plain sailing.... It's good to read your explanation... it paves the way.. –  Peter.O Apr 30 '11 at 0:45
    
I just (re-)discovered that Steven D's printf soultion won't handle the ASCII block of the unicode range, so your perl answer is now the best (for my particular requirements).. I'd previously ruled out printf (months ago), but I'd forgotten about it. Here is the queston/answer about its limits... Why does printf report an error on all but three (ASCII-range) Unicode Codepoints –  Peter.O May 2 '11 at 3:34
add comment

Bash 4.2 (released in 2011) added support for echo -e '\u0965'. printf %b '\u0965' and echo $'\u0965' also work.

http://tiswww.case.edu/php/chet/bash/FAQ:

o   $'...', echo, and printf understand \uXXXX and \UXXXXXXXX escape sequences.
share|improve this answer
    
Thanks... I'm still primarily using bash 4.1.5 in Ubuntu 10.04, but it's certainly good to know that it's now available in 4.2. (+1) –  Peter.O Mar 14 '13 at 13:56
1  
+1; note that bash 4.2.x versions have a bug where values between 0x80 and 0xff (128 - 255) - i.e., in the extended ASCII range - are NOT correctly UTF8-encoded and instead just passed through, resulting in an invalid UTF8 char that some terminals render as ?. As of (at least) 4.3.11 this has been fixed; if echo $'\ued' renders í, then the bug is not present. –  mklement0 May 7 at 5:15
add comment

If you have GNU coreutils, try printf:

$ printf '\u0965\n'
॥

echo can do the job if your console is using UTF-8 and you have the UTF-8 encoding:

$ echo -e '\xE0\xA5\xA5'

You can find a table of Unicode to UTF-8 hex encodings here: http://www.utf8-chartable.de/. You can convert the Unicode code points to hex using a number of scripting languages. Here is an example using python:

python -c "print(unichr(int('0965', 16)).encode('utf-8').encode('hex'))"

The following is a Perl script that will convert arguments to the correct hex value (many unnecessary parenthesis here):

#!/usr/bin/perl
use strict;
use warnings;
use 5.010;
use Encode;

foreach (@ARGV) {
    say unpack('H*', encode('utf8', chr(hex($_))))
}

For instance,

./uni2utf 0965
e0a5a5

Of course, if you have Perl or Python you could also just use those to print the characters.

share|improve this answer
    
Thanks.. The echo won't do what I want, as Codepoints are 2-byte UTF-16 Big-Endian.. but you've reminded me that there are 2 printf functions! (I thought printf could do it, and it seems that I ws invoking the wrong one)... $(which printf) works... Thanks for the python example.. but for this (my learning curve), I'm trying to stick as close as possible to "bash" as the only scriting language involved.. (when I'm comfortable enough with bash, I'll get stuck into Python... btw, .encode('hex') is one step beyond what I need.. (I thought it looked a bit busy in there :) –  Peter.O Apr 29 '11 at 8:02
    
Yeah, the .encode('hex') was just to get the hex code that seemed to work with echo for me. Glad that at least part of this was helpful. –  Steven D Apr 29 '11 at 8:07
    
I've just now seen you perl snippet.. thanks... it's good to have these various solutions tabled... The printf one is exactly what I was looking for (a single command, as per the zsh example) ..... I may a well post my not-using-another-scripting-language method which works on a stream of hex data (no \u, etc).. –  Peter.O Apr 29 '11 at 8:11
    
I particularly like the brevity of the printf above, but it doesnt handle values below ``\u00A0`... I've just re-discovered something I already knew (but dropped off the radar)... Here is a Question I asked about 4 months ago; Why does printf report an error on all but three (ASCII-range) Unicode Codepoints... So penguin359's perl solution is looking pretty good now :) .. It's a single invocaton, and I after "easy to type", so I'll give him the green-tick for perl –  Peter.O May 2 '11 at 3:21
add comment

UPDATE: Here is a bash way to do a single Unicode value ...(by "bash" I mean: not using any another scripting language) .. thanks to Gilles for a suggeston in this askubuntu Q/A .
According to this link: recode (Obsoletes iconv, dos2unix, unix2dos).. Edit: but as per the comment below, "obsoletes' may just mean "alternative"

      echo -n 0x0965 |recode UTF-16BE/x4..UTF-8

Here is a method to process a raw hex dump as input (ie. no escaped-prefixes like; \u0965, and no \x09\x65)..
xxd is a hex-dump utility (packaged with vim-common) which can revert a raw hex dump to the characters the dump represents... Unicode Codepoints are UTF-16BigEndian, which is exactly what a Hex-dump is..
xxd in revert mode accepts a stream of Hex values with line breaks.which are ignored.

This script creates a UTF-16BE stream, which it then reverts to the original chars.
The last line contains the two needed commands; xxd and iconv

for line in \
  "Matsuo Basho (1644-1694)" \
  "  pond" \
  "  frog jumps in" \
  "  plop!"
do 
  echo "$line" |iconv -f "$(locale charmap)" -t "UTF-16BE" |xxd -ps -u 
done |
#    (---this is the **revert** code---) 
tee >(xxd -p -u -r |iconv -f "UTF-16BE") ;echo

Here is the output (showing the UTF-16BE hex-dump input, first).
Note; xxd segments its own output with a newline at 60 hex-digits... The revert option ignores these newlines.. it ignores any/all newlines (as the aren't hex-digits)..

004D0061007400730075006F00200042006100730068006F002000280031
003600340034002D00310036003900340029000A
002000200070006F006E0064000A
0020002000660072006F00670020006A0075006D0070007300200069006E
000A
002000200070006C006F00700021000A

Matsuo Basho (1644-1694)
  pond
  frog jumps in
  plop!
share|improve this answer
    
Since it seems you used penguin359's information in your answer, you might consider marking his answer as correct rather than mine. –  Steven D Apr 29 '11 at 16:31
    
@Steven D: a noteworthy comment, but "seem" is the operative word. I've been using iconv like this for a couple of days now, which got me wondering if there is a single command. I've done similar entire-file processing in windows (C++), so I have a reasonlabe understanding of Unicode. I was really after a quick and simple bash method. By "bash" I mean: using the bash scripting language; not python/perl from within bash). I've added this as an answer because it may be of some value to someone reading this page. It is good one-liner for an entire file. Your printf is the best answer for me. –  Peter.O Apr 29 '11 at 16:59
1  
I wouldn't say recode obsoletes iconv, in fact recode is older than iconv, and these days iconv is far more commonly installed by default than recode (for example, on Linux, iconv is almost always installed because it comes with libc). –  Gilles May 2 '11 at 7:24
    
Thanks.. I was wondering about that.. That web page isn't exactly the definitive reference... so it is more of an alternative... –  Peter.O May 3 '11 at 2:15
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.