Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I tried, but I'm stuck at "escaping" the "sed's":

sed -i 's/\\x0/NUL/g' $1
sed -i 's/\\x1/SOH/g' $1
sed -i 's/\\x2/STX/g' $1
sed -i 's/\\x3/ETX/g' $1
sed -i 's/\\x4/EOT/g' $1
sed -i 's/\\x5/ENQ/g' $1
sed -i 's/\\x6/ACK/g' $1
sed -i 's/\\x7/BEL/g' $1
sed -i 's/\\x8/BS/g' $1
sed -i 's/\\x9/HT/g' $1
sed -i 's/\\x0A/LF/g' $1
sed -i 's/\\x0B/VT/g' $1
sed -i 's/\\x0C/FF/g' $1
sed -i 's/\\x0D/CR/g' $1
sed -i 's/\\x0E/SO/g' $1
sed -i 's/\\x0F/SI/g' $1
sed -i 's/\\x10/DLE/g' $1
sed -i 's/\\x11/DC1/g' $1
sed -i 's/\\x12/DC2/g' $1
sed -i 's/\\x13/DC3/g' $1
sed -i 's/\\x14/DC4/g' $1
sed -i 's/\\x15/NAK/g' $1
sed -i 's/\\x16/SYN/g' $1
sed -i 's/\\x17/ETB/g' $1
sed -i 's/\\x18/CAN/g' $1
sed -i 's/\\x19/EM/g' $1
sed -i 's/\\x1A/SUB/g' $1
sed -i 's/\\x1B/ESC/g' $1
sed -i 's/\\x1C/FS/g' $1
sed -i 's/\\x1D/GS/g' $1
sed -i 's/\\x1E/RS/g' $1
sed -i 's/\\x1F/US/g' $1
sed -i 's/\\x20//g' $1
sed -i 's/\\x21/!/g' $1
sed -i 's/\\x22/"/g' $1
sed -i 's/\\x23/#/g' $1
sed -i 's/\\x24/$/g' $1
sed -i 's/\\x25/%/g' $1
sed -i 's/\\x26/&/g' $1
sed -i 's/\\x27'/g' $1
sed -i 's/\\x28/(/g' $1
sed -i 's/\\x29/)/g' $1
sed -i 's/\\x2A/*/g' $1
sed -i 's/\\x2B/0/g' $1
sed -i 's/\\x2C/,/g' $1
sed -i 's/\\x2D/0/g' $1
sed -i 's/\\x2E/0/g' $1
sed -i 's/\\x2F///g' $1
sed -i 's/\\x30/0/g' $1
sed -i 's/\\x31/1/g' $1
sed -i 's/\\x32/2/g' $1
sed -i 's/\\x33/3/g' $1
sed -i 's/\\x34/4/g' $1
sed -i 's/\\x35/5/g' $1
sed -i 's/\\x36/6/g' $1
sed -i 's/\\x37/7/g' $1
sed -i 's/\\x38/8/g' $1
sed -i 's/\\x39/9/g' $1
sed -i 's/\\x3A/:/g' $1
sed -i 's/\\x3B/;/g' $1
sed -i 's/\\x3C/</g' $1
sed -i 's/\\x3D/=/g' $1
sed -i 's/\\x3E/>/g' $1
sed -i 's/\\x3F/?/g' $1
sed -i 's/\\x40/@/g' $1
sed -i 's/\\x41/A/g' $1
sed -i 's/\\x42/B/g' $1
sed -i 's/\\x43/C/g' $1
sed -i 's/\\x44/D/g' $1
sed -i 's/\\x45/E/g' $1
sed -i 's/\\x46/F/g' $1
sed -i 's/\\x47/G/g' $1
sed -i 's/\\x48/H/g' $1
sed -i 's/\\x49/I/g' $1
sed -i 's/\\x4A/J/g' $1
sed -i 's/\\x4B/K/g' $1
sed -i 's/\\x4C/L/g' $1
sed -i 's/\\x4D/M/g' $1
sed -i 's/\\x4E/N/g' $1
sed -i 's/\\x4F/O/g' $1
sed -i 's/\\x50/P/g' $1
sed -i 's/\\x51/Q/g' $1
sed -i 's/\\x52/R/g' $1
sed -i 's/\\x53/S/g' $1
sed -i 's/\\x54/T/g' $1
sed -i 's/\\x55/U/g' $1
sed -i 's/\\x56/V/g' $1
sed -i 's/\\x57/W/g' $1
sed -i 's/\\x58/X/g' $1
sed -i 's/\\x59/Y/g' $1
sed -i 's/\\x5A/Z/g' $1
sed -i 's/\\x5B/[/g' $1
sed -i 's/\\x5C/\/g' $1
sed -i 's/\\x5D/]/g' $1
sed -i 's/\\x5E/^/g' $1
sed -i 's/\\x5F/_/g' $1
sed -i 's/\\x60/`/g' $1
sed -i 's/\\x61/a/g' $1
sed -i 's/\\x62/b/g' $1
sed -i 's/\\x63/c/g' $1
sed -i 's/\\x64/d/g' $1
sed -i 's/\\x65/e/g' $1
sed -i 's/\\x66/f/g' $1
sed -i 's/\\x67/g/g' $1
sed -i 's/\\x68/h/g' $1
sed -i 's/\\x69/i/g' $1
sed -i 's/\\x6A/j/g' $1
sed -i 's/\\x6B/k/g' $1
sed -i 's/\\x6C/l/g' $1
sed -i 's/\\x6D/m/g' $1
sed -i 's/\\x6E/n/g' $1
sed -i 's/\\x6F/o/g' $1
sed -i 's/\\x70/p/g' $1
sed -i 's/\\x71/q/g' $1
sed -i 's/\\x72/r/g' $1
sed -i 's/\\x73/s/g' $1
sed -i 's/\\x74/t/g' $1
sed -i 's/\\x75/u/g' $1
sed -i 's/\\x76/v/g' $1
sed -i 's/\\x77/w/g' $1
sed -i 's/\\x78/x/g' $1
sed -i 's/\\x79/y/g' $1
sed -i 's/\\x7A/z/g' $1
sed -i 's/\\x7B/{/g' $1
sed -i 's/\\x7C/|/g' $1
sed -i 's/\\x7D/}/g' $1
sed -i 's/\\x7E/~/g' $1
sed -i 's/\\x7F//g' $1
sed -i 's/\\x80/€/g' $1
sed -i 's/\\x81//g' $1
sed -i 's/\\x82/‚/g' $1
sed -i 's/\\x83/ƒ/g' $1
sed -i 's/\\x84/„/g' $1
sed -i 's/\\x85/…/g' $1
sed -i 's/\\x86/†/g' $1
sed -i 's/\\x87/‡/g' $1
sed -i 's/\\x88/ˆ/g' $1
sed -i 's/\\x89/‰/g' $1
sed -i 's/\\x8A/Š/g' $1
sed -i 's/\\x8B/‹/g' $1
sed -i 's/\\x8C/Œ/g' $1
sed -i 's/\\x8D//g' $1
sed -i 's/\\x8E/Ž/g' $1
sed -i 's/\\x8F//g' $1
sed -i 's/\\x90//g' $1
sed -i 's/\\x91/‘/g' $1
sed -i 's/\\x92/’/g' $1
sed -i 's/\\x93/“/g' $1
sed -i 's/\\x94/”/g' $1
sed -i 's/\\x95/•/g' $1
sed -i 's/\\x96/–/g' $1
sed -i 's/\\x97/—/g' $1
sed -i 's/\\x98/˜/g' $1
sed -i 's/\\x99/™/g' $1
sed -i 's/\\x9A/š/g' $1
sed -i 's/\\x9B/›/g' $1
sed -i 's/\\x9C/œ/g' $1
sed -i 's/\\x9D//g' $1
sed -i 's/\\x9E/ž/g' $1
sed -i 's/\\x9F/Ÿ/g' $1
sed -i 's/\\xA0//g' $1
sed -i 's/\\xA1/¡/g' $1
sed -i 's/\\xA2/¢/g' $1
sed -i 's/\\xA3/£/g' $1
sed -i 's/\\xA4/¤/g' $1
sed -i 's/\\xA5/¥/g' $1
sed -i 's/\\xA6/¦/g' $1
sed -i 's/\\xA7/§/g' $1
sed -i 's/\\xA8/¨/g' $1
sed -i 's/\\xA9/©/g' $1
sed -i 's/\\xAA/ª/g' $1
sed -i 's/\\xAB/«/g' $1
sed -i 's/\\xAC/¬/g' $1
sed -i 's/\\xAD//g' $1
sed -i 's/\\xAE/®/g' $1
sed -i 's/\\xAF/¯/g' $1
sed -i 's/\\xB0/°/g' $1
sed -i 's/\\xB1/±/g' $1
sed -i 's/\\xB2/²/g' $1
sed -i 's/\\xB3/³/g' $1
sed -i 's/\\xB4/´/g' $1
sed -i 's/\\xB5/µ/g' $1
sed -i 's/\\xB6/¶/g' $1
sed -i 's/\\xB7/·/g' $1
sed -i 's/\\xB8/¸/g' $1
sed -i 's/\\xB9/¹/g' $1
sed -i 's/\\xBA/º/g' $1
sed -i 's/\\xBB/»/g' $1
sed -i 's/\\xBC/¼/g' $1
sed -i 's/\\xBD/½/g' $1
sed -i 's/\\xBE/¾/g' $1
sed -i 's/\\xBF/¿/g' $1
sed -i 's/\\xC0/À/g' $1
sed -i 's/\\xC1/A/g' $1
sed -i 's/\\xC2/Â/g' $1
sed -i 's/\\xC3/Ã/g' $1
sed -i 's/\\xC4/Ä/g' $1
sed -i 's/\\xC5/Å/g' $1
sed -i 's/\\xC6/Æ/g' $1
sed -i 's/\\xC7/Ç/g' $1
sed -i 's/\\xC8/È/g' $1
sed -i 's/\\xC9/E/g' $1
sed -i 's/\\xCA/Ê/g' $1
sed -i 's/\\xCB/Ë/g' $1
sed -i 's/\\xCC/Ì/g' $1
sed -i 's/\\xCD/I/g' $1
sed -i 's/\\xCE/Î/g' $1
sed -i 's/\\xCF/Ï/g' $1
sed -i 's/\\xD0/Ð/g' $1
sed -i 's/\\xD1/Ñ/g' $1
sed -i 's/\\xD2/Ò/g' $1
sed -i 's/\\xD3/O/g' $1
sed -i 's/\\xD4/Ô/g' $1
sed -i 's/\\xD5/Õ/g' $1
sed -i 's/\\xD6/O/g' $1
sed -i 's/\\xD7/×/g' $1
sed -i 's/\\xD8/Ø/g' $1
sed -i 's/\\xD9/Ù/g' $1
sed -i 's/\\xDA/U/g' $1
sed -i 's/\\xDB/Û/g' $1
sed -i 's/\\xDC/U/g' $1
sed -i 's/\\xDD/Ý/g' $1
sed -i 's/\\xDE/Þ/g' $1
sed -i 's/\\xDF/ß/g' $1
sed -i 's/\\xE0/à/g' $1
sed -i 's/\\xE1/a/g' $1
sed -i 's/\\xE2/â/g' $1
sed -i 's/\\xE3/ã/g' $1
sed -i 's/\\xE4/ä/g' $1
sed -i 's/\\xE5/å/g' $1
sed -i 's/\\xE6/æ/g' $1
sed -i 's/\\xE7/ç/g' $1
sed -i 's/\\xE8/è/g' $1
sed -i 's/\\xE9/e/g' $1
sed -i 's/\\xEA/ê/g' $1
sed -i 's/\\xEB/ë/g' $1
sed -i 's/\\xEC/ì/g' $1
sed -i 's/\\xED/i/g' $1
sed -i 's/\\xEE/î/g' $1
sed -i 's/\\xEF/ï/g' $1
sed -i 's/\\xF0/ð/g' $1
sed -i 's/\\xF1/ñ/g' $1
sed -i 's/\\xF2/ò/g' $1
sed -i 's/\\xF3/o/g' $1
sed -i 's/\\xF4/ô/g' $1
sed -i 's/\\xF5/õ/g' $1
sed -i 's/\\xF6/o/g' $1
sed -i 's/\\xF7/÷/g' $1
sed -i 's/\\xF8/ø/g' $1
sed -i 's/\\xF9/ù/g' $1
sed -i 's/\\xFA/u/g' $1
sed -i 's/\\xFB/û/g' $1
sed -i 's/\\xFC/u/g' $1
sed -i 's/\\xFD/ý/g' $1
sed -i 's/\\xFE/þ/g' $1
sed -i 's/\\xFF/\ÿ/g' $1

How can I create a hex-code TO normal char converter? So that if I run:

sh thementionedscript.sh hexstrings.txt

-->> all the HEX chars in hexstrings.txt should be converted to normal chars, like: before:

\x68\x69

after:

hi
share|improve this question
2  
By the way, your example script is converting from \xXX to an encoding which is not the normal UTF-8, as there is no such thing as a single byte UTF-8 character > \x7F... If this is what you are actually after, then the current answers work only if the locale is appropriately set.... It looks like you want a conversion to Windows-1252 Character Encoding –  Peter.O Oct 22 '11 at 23:06

4 Answers 4

In bash:

#!/bin/bash

while IFS= read -r line; do
    printf "$line"
done

IFS= is needed to stop IFS stripping occurring, that is, in most cases (where IFS contains whitespace, which is the norm) if you don't make IFS empty, whitespace will be stripped from each side of "${line}" regardless of the fact that you have quoted it. The -r option to read stops backspace interpretation.

Bear in mind that this will also interpret other escape codes which may be present in your source. If this concerns you, read the file character by character and only have printf act on \x escape codes.

You should always use a tempfile, so run it like this:

./foo < myfile > mytempfile && mv mytempfile myfile

Also, do not use sed -i to edit files, it is not portable, and even where it is the syntax varies between implementations (your code will fail on Mac OSX and most BSDs, for example). ed is the better option to edit files.

share|improve this answer
    
What? You're back too? Time for me to have a break then... –  rozcietrzewiacz Oct 22 '11 at 20:28
    
Oh come on, who uses ed? It's just too magic ;P –  rozcietrzewiacz Oct 22 '11 at 20:30
2  
ed is POSIX, so it's far superior to other non-POSIX methods. And I use it, it still has one friend left ;) –  Chris Down Oct 22 '11 at 20:33
2  
+1 from me. "ed, man! !man ed" FTW. –  jw013 Oct 22 '11 at 20:39
    
Ah, Darn you both. I'll learn it. –  rozcietrzewiacz Oct 22 '11 at 21:07

In bash?

#!/bin/bash

while IFS= read line
 do
 echo -e "$line"
done
share|improve this answer

Here is a new take on the question...

To use sed to make the changes, you are 97% of the way there.
Here are some essential and some optional points to get it to work.

  • It may be useful to make all you patterns case insensitive. Use ig instead of g

  • You can make all replacements in one pass. Simply concatenate them, eg. 's/\\x00/NUL/ig;s/\\x01/SOH/ig;s/\\x02/STX/ig;...',
    or
    's/\\x01/SOH/ig; 
      s/\\x02/STX/ig; 
      ... 
      s/\\xFE/þ/ig; 
      s/\\xFF/ÿ/ig;'

  • You need to change some of the replacement expressions:
    /'/ should be /\x27/     It clashes with the quoting ' in 's/a/b/ig'
    /\/ should be /\\/        Otherwise it escapes the /
    /&/ should be /\&/        As on its own, it is a backreference

  • You need to switch the / delimiter when / occurrs in your replacement expression.. Just for this one expression, change it to | (or whatever takes your fancy), eg. s|\\x2F|/|g; Some sed situations require that it be escaped at the start of a pattern, eg. a range expression. '\|^abc|q'

  • I've noticed a few differences in your list compared to what I came up with (based on the encoding being Windows-1252). You may like to check these: 7F('DEL')   A0(' ')   AD('SHY')

  • I encounter varying reports on the way sed -i handles so-called insitu updates. It uses a temp file in any case, so I lean towards using sponge (package moreutils) for rewriting the text file..

That's about it. Here is an example of the resulting sed command...

 sed -e '
 s/\\x01/SOH/ig;
 s/\\x02/STX/ig;
 ...
 s/\\xFE/þ/ig;
 s/\\xFF/ÿ/ig;
 ' "$1" | sponge "$1"

The full command can be found at paste.ubuntu

share|improve this answer
    
sponge isn't standard (I'd never heard the term in some 25 years of Unix, in fact). –  vonbrand Jan 26 '13 at 2:51

UPDATE If you want to interpret the hex values as Windows-1252 and represent them in your current locale's encoding (typically UTF-8 for *nix), you can use the following command. Note. That aside from the newlines (\n) for each input line, this snippet takes as input only data in the form \x33\x66\x99\xBB\xFF... anything else causes an error.

<file sed -re 's|(.)\\x|\1,0x|g; s|^\\x|0x|; $!{s|$|,0x0a|}' |
 tr '\n' ',' | recode Windows-1252/x1..UTF-8 

From here on, is my original answer (which doesn't consider diferent encodings)

You can use xxd in revert (-r) postscript (-p) mode. xxd is part of the vi package.
Note. The test data contains hexed \n (\x0a) chars.
xxd ignores any real \n chars, so you split the hex data over as many lines you wish (as in the test file).

xxd is also useful for converting a straight hexdump, eg. 68656c6c6f20776f726c64 (hello world)... sed is used in this example purely to remove the `\x'.

This first snippet ignores chars which are not in \xXX format...

<file sed -e 's|\\x||g' | xxd -p -r

If you want to keep the newlines (\n vs \x0a) from the souce file, use this (other text is ignored)

<file sed -e 's|\\x||g; s|$|0a|' | xxd -p -r

create a test file

echo -n "
\x63\x61\x74\x0a\x64\x6f\x67\x0a\x62\x69\x72\x64\x0a\x6d\x6f\x6f
\x73\x65\x0a\x61\x6c\x62\x61\x74\x72\x6f\x73\x73" >file

output (from 1st snippet, which only acts on \xXX data)

cat
dog
bird
moose
albatross
share|improve this answer
    
Bear in mind that this won't work with most standard files that aren't just a bunch of hex. Take the following as an example: the raw file i would like a \x63\x61\x74\x0a or maybe a \x64\x6f\x67\x0a would have no output. –  Chris Down Oct 22 '11 at 21:49
    
?? That is exactly what it does. Have you tested it? The test file contains exactly what is shown in the echo statement.. or do you mean it won't keep the newlines in the source file? ... that's a good point if that's what you mean.. (easily fixed. ammended answer).. –  Peter.O Oct 22 '11 at 22:15
    
@Chris: I finally got your meaning (I was bleary eyed when I first looked at it)... yes, thanks, I was aware of that, but as the question has no reference to anything other than \xXX , I posted this answer, as it is a pretty simple way to convert without a loop.. –  Peter.O Oct 23 '11 at 2:28
    
I read the OP as referring to content which contained hex character codes, but not necessarily in full. –  Chris Down Oct 23 '11 at 9:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.