10

I've got a html file with a lot of %-encoded UTF-8 text in URLs.

For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian).

The task is to replace all such substrings with readable UTF-8 text.

To simplify the task we can consider there is no other % sign usage in the file. Letter digits can be both upper and lower case.

I suspect this can be done elegantly with sed, perl, awk or something but don't know how.

This web application seems to do the trick with text you paste there.

0

3 Answers 3

10

With bash, zsh, GNU echo or some implementations of ksh on some systems, this can be decoded simply by echo -e after replacing all % with \x.

url_encoded_string="%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B"
temp_string=${url_encoded_string//%/\\x}

printf '%s\n' "$temp_string"
# output: \xD1\x80\xD0\xB5\xD1\x81\xD1\x83\xD1\x80\xD1\x81\xD1\x8B

echo -e "$temp_string"
# output: ресурсы

(It assumes the string itself doesn't contain backslash characters and is not one of the options supported by your echo command)

As @JoshLee also points out, the "echo caveat" can be avoided by directly using:

printf ${url_encoded_string//%/\\x}

instead directly behind the first command.

1
  • Note that this elegant solution will work with any encoding, not just UTF-8 (i.e., get rid of the encodings for ~ and others. Another trick to add to my toolbox. Thanks!
    – vonbrand
    Commented Jan 26, 2013 at 2:37
5

With perl:

perl -pe 's/%([0-9A-F]{2})/pack"H2",$1/gei'

Or with URI::Escape:

perl -MURI::Escape -pe '$_=uri_unescape$_'
2
  • I love this because I can pipe it whatever I want thanks to $_ gnu.org/software/bash/manual/html_node/Special-Parameters.html
    – Nemo
    Commented May 22, 2018 at 8:54
  • @Nemo, $_ here is perl's $_, not bash's. In combination with the -p option, the perl expression is run for every input record (records being read from files given as argument or stdin if no argument is provided), with the current record stored in $_. That's similar to awk's $0. Commented May 22, 2018 at 8:58
0

There is a program called convmv which can help you.

Simply use convmv --unescape /some_path/target_file. It will do a dry-run.

Once you have confirmed, use convmv --notest --unescape /some_path/target_file to continue.

The homepage of this program is: http://j3e.de/linux/convmv/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .