9

I've got a html file with a lot of %-encoded UTF-8 text in URLs.

For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian).

The task is to replace all such substrings with readable UTF-8 text.

To simplify the task we can consider there is no other % sign usage in the file. Letter digits can be both upper and lower case.

I suspect this can be done elegantly with sed, perl, awk or something but don't know how.

This web application seems to do the trick with text you paste there.

9

With bash, zsh, GNU echo or some implementations of ksh on some systems, this can be decoded simply by echo -e after replacing all % with \x.

url_encoded_string="%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B"
temp_string=${url_encoded_string//%/\\x}

printf '%s\n' "$temp_string"
# output: \xD1\x80\xD0\xB5\xD1\x81\xD1\x83\xD1\x80\xD1\x81\xD1\x8B

echo -e "$temp_string"
# output: ресурсы

(It assumes the string itself doesn't contain backslash characters and is not one of the options supported by your echo command)

As @JoshLee also points out, the "echo caveat" can be avoided by directly using:

printf ${url_encoded_string//%/\\x}

instead directly behind the first command.

  • Note that this elegant solution will work with any encoding, not just UTF-8 (i.e., get rid of the encodings for ~ and others. Another trick to add to my toolbox. Thanks! – vonbrand Jan 26 '13 at 2:37
5

With perl:

perl -pe 's/%([0-9A-F]{2})/pack"H2",$1/gei'

Or with URI::Escape:

perl -MURI::Escape -pe '$_=uri_unescape$_'
  • I love this because I can pipe it whatever I want thanks to $_ gnu.org/software/bash/manual/html_node/Special-Parameters.html – Nemo May 22 '18 at 8:54
  • @Nemo, $_ here is perl's $_, not bash's. In combination with the -p option, the perl expression is run for every input record (records being read from files given as argument or stdin if no argument is provided), with the current record stored in $_. That's similar to awk's $0. – Stéphane Chazelas May 22 '18 at 8:58
0

There is a program called convmv which can help you.

Simply use convmv --unescape /some_path/target_file. It will do a dry-run.

Once you have confirmed, use convmv --notest --unescape /some_path/target_file to continue.

The homepage of this program is: http://j3e.de/linux/convmv/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.