I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this?

I did search a bit through unix.stackexchange.com and on the internet but I couldn't find any command line tool for decoding url encoding.

What I want to do is simply in place edit a txt file so that:

  • %21 becomes !
  • %23 becomes #
  • %24 becomes $
  • %26 becomes &
  • %27 becomes '
  • %28 becomes (
  • %29 becomes )

And so on.

12 Answers 12

up vote 85 down vote accepted

Found these Python one liners that do what you want:

$ alias urldecode='python -c "import sys, urllib as ul; \
    print ul.unquote_plus(sys.argv[1])"'

$ alias urlencode='python -c "import sys, urllib as ul; \
    print ul.quote_plus(sys.argv[1])"'

Example

$ urldecode 'q+werty%3D%2F%3B'
q werty=/;

$ urlencode 'q werty=/;'
q+werty%3D%2F%3B

References

  • I know this very late, but is there any way i can do this with in place editing? – DisplayName Nov 4 '14 at 11:18
  • @DisplayName - sounds like a new Q to me. I'd ask it, and reference this one. – slm Nov 4 '14 at 12:18
  • 12
    streaming: cat your_lovely_file.csv| python -c "import sys, urllib as ul; [sys.stdout.write(ul.quote_plus(l)) for l in sys.stdin]" – kirill_igum Dec 2 '15 at 15:14
  • 4
    Note, that this is a Python 2; on the systems where python is 3 by default, this will result in error. Changing python to python2 helps. – Ivan Kolmychek Dec 10 '15 at 15:02
  • For python3 you can use import urllib.parse as ul instead of import urllib as ul. – ibotty Nov 20 '17 at 8:25

sed

Try the following command line:

$ sed 's@+@ @g;s@%@\\x@g' file | xargs -0 printf "%b"

or the following alternative using echo -e:

$ sed -e's/%\([0-9A-F][0-9A-F]\)/\\\\\x\1/g' file | xargs echo -e

Note: The above syntax may not convert + to spaces, and can eat all the newlines.


You may define it as alias and add it to your shell rc files:

$ alias urldecode='sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b"'

Then every time when you need it, simply go with:

$ echo "http%3A%2F%2Fwww" | urldecode
http://www

bash

When scripting, you can use the following syntax:

input="http%3A%2F%2Fwww"
decoded=$(printf '%b' "${input//%/\\x}")

However above syntax won't handle pluses (+) correctly, so you've to replace them with spaces via sed.

You can also use the following urlencode() and urldecode() functions:

urlencode() {
    # urlencode <string>
    local length="${#1}"
    for (( i = 0; i < length; i++ )); do
        local c="${1:i:1}"
        case $c in
            [a-zA-Z0-9.~_-]) printf "$c" ;;
            *) printf '%%%02X' "'$c"
        esac
    done
}

urldecode() {
    # urldecode <string>

    local url_encoded="${1//+/ }"
    printf '%b' "${url_encoded//%/\\x}"
}

Note that your urldecode() assumes the data contains no backslash.


bash + xxd

Bash function with xxd tool:

urlencode() {
  local length="${#1}"
  for (( i = 0; i < length; i++ )); do
    local c="${1:i:1}"
    case $c in
      [a-zA-Z0-9.~_-]) printf "$c" ;;
    *) printf "$c" | xxd -p -c1 | while read x;do printf "%%%s" "$x";done
  esac
done
}

Found in cdown's gist file, also at stackoverflow.


PHP

Using PHP you can try the following command:

$ echo oil+and+gas | php -r 'echo urldecode(fgets(STDIN));' // Or: php://stdin
oil and gas

or just:

php -r 'echo urldecode("oil+and+gas");'

Use -R for multiple line input.


Perl

In Perl you can use URI::Escape.

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

Or to process a file:

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

awk

Try anon solution:

awk -niord '{printf RT?$0chr("0x"substr(RT,2)):$0}' RS=%..

Note: Parameter -n is specific to GNU awk.

See: Using awk printf to urldecode text.

decoding file names

If you need to remove url encoding from the file names, use deurlname tool from renameutils (e.g. deurlname *.*).

See also:


Related:

  • 1
    awk: As this makes use of a library function, chr(), there is a high probability it will solely work on GNU awk (gawk). However, in this case there will be hardly any equivalent for POSIX awk, because the -n option (allowing non-decimal arguments) IS a GNU awk specialty. – syntaxerror Jun 27 '15 at 15:07
  • Your first sed code gives me xargs: argument line too long for a file with ≥2164 lines. – Sparhawk Aug 30 '16 at 5:15
  • @Sparhawk Try the second sed example, or try adding -L1 to xargs to split the lines being read. For more complex input, you should avoid xargs or piping in the shell, but try parsing the file directly, like using php. – kenorb Aug 30 '16 at 8:33
  • 2
    Your solutions involving printf do not take into account that the url might contain escaped percentage signs like %25. You pass these to printf without escaping them for printf with another percentage sign like %%. – josch Sep 13 '16 at 19:57
  • 1
    The bash version requires local LC_ALL=C at the top, otherwise all wide characters (i.e Japanese, Chinese, etc.) are not properly broken into bytes. – Phernost Aug 27 at 16:31

There is a built-in function for that in the Python standard library. In Python 2, it's urllib.unquote.

decoded_url=$(python2 -c 'import sys, urllib; print urllib.unquote(sys.argv[1])' "$encoded_url")

Or to process a file:

python2 -c 'import sys, urllib; print urllib.unquote(sys.stdin.read())' <file >file.new &&
mv -f file.new file

In Python 3, it's urllib.parse.unquote.

decoded_url=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.unquote(sys.argv[1]))' "$encoded_url")

Or to process a file:

python3 -c 'import sys, urllib; print(urllib.parse.unquote(sys.stdin.read()))' <file >file.new &&
mv -f file.new file

In Perl you can use URI::Escape.

decoded_url=$(perl -MURI::Escape -e 'print uri_unescape($ARGV[0])' "$encoded_url")

Or to process a file:

perl -i -MURI::Escape -e 'print uri_unescape($ARGV[0])' file

If you want to stick to POSIX portable tools, it's awkward, because the only serious candidate is awk, which doesn't parse hexadecimal numbers. See Using awk printf to urldecode text for examples with common awk implementations, including BusyBox.

If you want to use a simple-minded sed command, then use the following:

sed -e 's/%21/!/g' -e 's/%23/#/g' -e 's/%24/$/g' -e 's/%26/\&/g' -e "s/%27/'/g" -e 's/%28/(/g' -e 's/%29/)/g'

But it is more convenient to create a script like (say sedscript):

s/%21/!/g
s/%23/#/g
s/%24/$/g
s/%26/\&/g
s/%27/'/g
s/%28/(/g
s/%29/)/g

Then run sed -f sedscript < old > new, which will output as you desired.


For an ease, the command urlencode is also available directly in gridsite-clients package can be installed from (by sudo apt-get install gridsite-clients in Ubuntu/Debian system).

NAME

    urlencode - convert strings to or from URL-encoded form
SYNOPSIS

    urlencode [-m|-d] string [string ...]

DESCRIPTION

    urlencode encodes strings according to RFC 1738.

    That is, characters A-Z a-z 0-9 . _ and - are passed through unmodified, but all other characters are represented as %HH, where HH is their two-digit upper-case hexadecimal ASCII representation. For example, the URL http://www.gridpp.ac.uk/ becomes http%3A%2F%2Fwww.gridpp.ac.uk%2F

    urlencode converts each character in all the strings given on the command line.  If multiple strings are given, they are concatenated with separating spaces before conversion.

OPTIONS
    -m
      Instead of full conversion, do GridSite "mild URL encoding" in which A-Z a-z 0-9 . = - _ @ and / are passed through unmodified.  This results in slightly more human-readable strings but the application must be prepared to create or simulate the directories implied by any slashes.
    -d
      Do URL-decoding rather than encoding, according to RFC 1738.  %HH and %hh strings are converted and other characters are passed through unmodified, with the exception that + is converted to space.

Example of decoding URL:

$ urlencode -d "http%3a%2f%2funix.stackexchange.com%2f"
http://unix.stackexchange.com/

$ urlencode -d "Example: %21, %22, . . . , %29 etc"
Example: !, ", . . . , ) etc
  • For tutorial on sed visit – Pandya Oct 4 '14 at 14:27
  • 4
    This is a bad solution, because it requires hardcoding every character. This problem is exemplified by your code missing the often used %20 escape sequence. – Overv Oct 5 '14 at 13:43
  • @Overv I've just Revised – Pandya Oct 5 '14 at 14:02
  • Also, you might want to double-check what s/%26/&/g does.  (I fixed it.) – G-Man Jul 16 '15 at 6:42

GNU awk

#!/usr/bin/awk -fn
@include "ord"
BEGIN {
  RS = "%.."
}
{
  printf RT ? $0 chr("0x" substr(RT, 2)) : $0
}

Or

#!/bin/sh
awk -niord '{printf RT?$0chr("0x"substr(RT,2)):$0}' RS=%..

Convert percent-encoded file URL to local file in bash

I can't comment on best answer in this thread, so here is mine.

Personally, I use these aliases for URL encoding and decoding:

alias urlencode='python -c "import urllib, sys; print urllib.quote(  sys.argv[1] if len(sys.argv) > 1 else sys.stdin.read()[0:-1])"'

alias urldecode='python -c "import urllib, sys; print urllib.unquote(sys.argv[1] if len(sys.argv) > 1 else sys.stdin.read()[0:-1])"'

Both commands allow you to convert data, passed as a command line argument or read it from standard input, because both one-liners check whether there are command line arguments (even empty ones) and process them or just read standard input otherwise.


update 2017-05-23 (slash encoding)

In response to the @Bevor's comment.

If you also need to encode the slash, just add an empty second argument to the quote function, then the slash will also be encoded.

So, finally urlencode alias in bash looks like this:

alias urlencode='python -c "import urllib, sys; print urllib.quote(sys.argv[1] if len(sys.argv) > 1 else sys.stdin.read()[0:-1], \"\")"'

Example

$ urlencode "Проба пера/Pen test"
%D0%9F%D1%80%D0%BE%D0%B1%D0%B0%20%D0%BF%D0%B5%D1%80%D0%B0%2FPen%20test

$ echo "Проба пера/Pen test" | urlencode
%D0%9F%D1%80%D0%BE%D0%B1%D0%B0%20%D0%BF%D0%B5%D1%80%D0%B0%2FPen%20test

$ urldecode %D0%9F%D1%80%D0%BE%D0%B1%D0%B0%20%D0%BF%D0%B5%D1%80%D0%B0%2FPen%20test
Проба пера/Pen test

$ echo "%D0%9F%D1%80%D0%BE%D0%B1%D0%B0%20%D0%BF%D0%B5%D1%80%D0%B0%2FPen%20test" | urldecode
Проба пера/Pen test

$ urlencode "Проба пера/Pen test" | urldecode
Проба пера/Pen test

$ echo "Проба пера/Pen test" | urlencode | urldecode
Проба пера/Pen test
  • 1
    Does not encode slashes. – Bevor May 15 '17 at 11:41
  • @Bevor: Example? – DIG mbl May 22 '17 at 3:40
  • Add a slash to urlencode "Проба пера" -> result: Slash is not encoded. – Bevor May 22 '17 at 9:21
  • 1
    @Bevor: You are right. Thank you for your comment. I will also change my answer to reflect your comment in it. – DIG mbl May 23 '17 at 13:49

Perl one liner:

$ perl -pe 's/\%(\w\w)/chr hex $1/ge'

Example:

$ echo '%21%22' |  perl -pe 's/\%(\w\w)/chr hex $1/ge'
!"
  • 1
    This answer is attractive when you don't want to deal with installing perl modules. – user7000 Nov 10 '15 at 19:36

And another Perl approach:

#!/usr/bin/env perl
use URI::Encode;
my $uri     = URI::Encode->new( { encode_reserved => 0 } );
while (<>) {

    print $uri->decode($_)
}

You will need to install the URI::Encode module. On my Debian, I could simply run

sudo apt-get install liburi-encode-perl

Then, I ran the script above on a test file containing:

http://foo%21asd%23asd%24%26asd%27asd%28asd%29

The result was (I had saved the script as foo.pl):

$ ./foo.pl
http://foo!asd#asd$&asd'asd(asd)

An answer in (mostly Posix) shell:

$ input='%21%22'
$ printf "`printf "%s\n" "$input" | sed -e 's/+/ /g' -e 's/%\(..\)/\\\\x\1/g'`"
!"

Explanation:

  • -e 's/+/ /g transforms each + in space (as described in url-encode norm)
  • -e 's/%\(..\)/\\\\x\1/g' transform each %XX in \\xXX. Notice one of \ will be removed by quoting rules.
  • The inner printf is just there to pass input to sed. We may replace it by any other mechanism
  • The outer printf interpret \\xXX sequences and display result.

Edit:

Since % should always been interpreted in URLs, it is possible to simplify this answer. In add, I think it is cleaner to use xargs instead of backquotes (thanks to @josch).

$ input='%21%22+%25'
$ printf "%s\n" "$input" | sed -e 's/+/ /g; s/%/\\x/g' | xargs -0 printf
!" %

Unfortunately, (as @josch noticed) none of these solutions are Posix compliant since \x escape sequence is not defined in Posix.

  • Welcome to U&L. Perhaps you could explain this answer and how it works. We generally prefer our answers to be long form with details, not just code snippets. – slm Dec 5 '14 at 12:41
  • I really like this answer because it's comprehensive, portable and doesn't require extra heavier external programs like perl or python. Works well for me. – Steve Wills Jan 15 '15 at 17:05
  • 1
    Great solution. And even shorter and smarter: ... | sed 's/+/ /g;s/%\(..\)/\\\\x\1/g'. The -e option can be omitted here in fact... – syntaxerror Jun 27 '15 at 16:09
  • In which Posix shell can you confirm your solution to work? If I execute your statements, then I get this printed: \x21\x22. A fix that works for me is to replace the printf "printf ... | sed ..." construct by printf ... | sed ... | xargs printf. I assume the problem is with how characters get expanded in my shell when passing them through a doubly quoted string to the outer printf. I'm using dash on Debian as my Posix shell. – josch Sep 13 '16 at 19:54
  • 1
    @josch You are right, printf is a built-in in dash and it doesn't recognize \x escaping. You can use /usr/bin/printf instead of printf in order to make it work. Normally, you should be able to use command printf, but it seems to not work as it should. It continue to use built-in. – Jezz Sep 14 '16 at 11:43

Shell-only:

$ x='a%20%25%e3%81%82';printf "${x//\%/\\x}"
a %あ

Add -- or %b to prevent arguments that start with a dash from being treated as options.

In zsh ${x//%/a} adds a to the end but ${x//\%/a} replaces % with a.

Here are the relevant bits from another script (that I just shamelessly stole from my youtube.com download script from another answer) I've written before. It uses sed and the shell to build up a working urldecode.

set \! \" \# \$ \% \& \' \( \) \* \ \+ \, \/ \: \; \= \? \@ \[ \]
for c do set "$@" "'$c" "$c"; shift; done
curl -s "$url" | sed 's/\\u0026/\&/g;'"$(
    printf 's/%%%X/\\%s/g;' "$@"
)"

I wont swear it's comprehensive - and in fact I doubt it - but it handled youtube surely enough.

Here is a BASH function to do exactly that:

function urldecode() {
        echo -ne $(echo -n "$1" | sed -E "s/%/\\\\x/g")
}

helped me. hope it will help you.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.