Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've got a html file with a lot of %-encoded UTF-8 text in URLs.

For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian).

The task is to replace all such substrings with readable UTF-8 text.

To simplify the task we can consider there is no other % sign usage in the file. Letter digits can be both upper and lower case.

I suspect this can be done elegantly with sed, perl, awk or something but don't know how.

This web application seems to do the trick with text you paste there.

share|improve this question
add comment

3 Answers

up vote 8 down vote accepted

With bash, zsh, GNU echo or some implementations of ksh on some systems, this can be decoded simply by echo -e after replacing all % with \x.

url_encoded_string="%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B"
temp_string=${url_encoded_string//%/\\x}

printf '%s\n' "$temp_string"
# output: \xD1\x80\xD0\xB5\xD1\x81\xD1\x83\xD1\x80\xD1\x81\xD1\x8B

echo -e "$temp_string"
# output: ресурсы

(It assumes the string itself doesn't contain backslash characters and is not one of the options supported by your echo command)

share|improve this answer
1  
Or simply: printf ${url_encoded_string//%/\\x} –  Josh Lee Dec 11 '12 at 20:05
    
Note that this elegant solution will work with any encoding, not just UTF-8 (i.e., get rid of the encodings for ~ and others. Another trick to add to my toolbox. Thanks! –  vonbrand Jan 26 '13 at 2:37
add comment

There is a program called convmv which can help you.

Simply use convmv --unescape /some_path/target_file. It will do a dry-run.

Once you have confirmed, use convmv --notest --unescape /some_path/target_file to continue.

The homepage of this program is: http://j3e.de/linux/convmv/

share|improve this answer
add comment

With perl:

perl -pe 's/%([0-9A-F]{2})/pack"H2",$1/gei'

Or with URI::Escape:

perl -MURI::Escape -pe '$_=uri_unescape$_'
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.