How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

Question

I've got a html file with a lot of %-encoded UTF-8 text in URLs.

For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian).

The task is to replace all such substrings with readable UTF-8 text.

To simplify the task we can consider there is no other % sign usage in the file. Letter digits can be both upper and lower case.

I suspect this can be done elegantly with sed, perl, awk or something but don't know how.

This web application seems to do the trick with text you paste there.

Answer 1

With bash, zsh, GNU echo or some implementations of ksh on some systems, this can be decoded simply by echo -e after replacing all % with \x.

url_encoded_string="%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B"
temp_string=${url_encoded_string//%/\\x}

printf '%s\n' "$temp_string"
# output: \xD1\x80\xD0\xB5\xD1\x81\xD1\x83\xD1\x80\xD1\x81\xD1\x8B

echo -e "$temp_string"
# output: ресурсы

(It assumes the string itself doesn't contain backslash characters and is not one of the options supported by your echo command)

As @JoshLee also points out, the "echo caveat" can be avoided by directly using:

printf ${url_encoded_string//%/\\x}

instead directly behind the first command.

Answer 2

With perl:

perl -pe 's/%([0-9A-F]{2})/pack"H2",$1/gei'

Or with URI::Escape:

perl -MURI::Escape -pe '$_=uri_unescape$_'

Answer 3

There is a program called convmv which can help you.

Simply use convmv --unescape /some_path/target_file. It will do a dry-run.

Once you have confirmed, use convmv --notest --unescape /some_path/target_file to continue.

The homepage of this program is: http://j3e.de/linux/convmv/