Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need a way to URL encode strings with a shell script on a OpenWRT device running old version of busybox. Right now I ended up with the following code:

urlencode() {
echo "$@" | awk -v ORS="" '{ gsub(/./,"&\n") ; print }' | while read l
do
  c="`echo "$l" | grep '[^-._~0-9a-zA-Z]'`"
  if [ "$l" == "" ]
  then
    echo -n "%20"
  else
    if [ -z "$c" ]
    then
      echo -n "$l"
    else
      printf %%%02X \'"$c"
    fi
  fi
done
echo ""
}

This works more or less fine but there're a few flaws:

  1. Some characters are skipped, like "\", for example.
  2. Result is return character by character so it's extremely slow. It takes about 20 seconds to url encode just a few strings inside a batch.

I'm not very much into shell scripting in obscure shells (ash here) so I summon to collective wisdom to help me improve it so it works faster and doesn't skip any characters. I would appreciate an advice but please don't offer substitution algorithms as I'm looking to support all characters not just a few special ones. Also, my version of bash doesn't support substring like this ${var:x:y}.

Thanks!

share|improve this question
    
There is a small set of characters that are allowed in URLs. Everything else must be encoded. You may be better off having only two branches, one for allowed characters, the other to escape, using some simple way to get the hex value of the character (the encoding is not dark magic, it's just the hex value, %20 is 0x20 which is a space in ASCII and UTF-8). See stackoverflow.com/a/10660730/1079308 –  njsg Jan 8 '13 at 19:20
    
Thanks! Reducing the number of if branches speeded up the encoding dramatically to the point it's sufficient for my use case. However, I'm still unable to encode certain characters like "$$" or "\". Would really appreciate an advice on how to escape the content of the variable properly. –  Serg Jan 8 '13 at 21:46
    
Note that I've updated my answer: my first versions were all forking an external [ utility on my router. Avoiding the external [ is a significant speed improvement. –  Gilles Jan 9 '13 at 11:29
add comment

1 Answer 1

up vote 3 down vote accepted

[TL,DR: use the urlencode_grouped_case version in the last code block.]

Awk can do most of the job, except that it annoyingly lacks a way to convert from a character to its number. If od is present on your device, you can use it to convert all characters (more precisely, bytes) into the corresponding number (written in decimal, so that awk can read it), then use awk to convert valid characters back into literals and quoted characters into the proper form.

urlencode_od_awk () {
  echo "$1" | od -t d1 | awk '{
      for (i = 2; i <= NF; i++) {
        printf(($i>=48 && $i<=57) || ($i>=65 &&$i<=90) || ($i>=97 && $i<=122) ||
                $i==45 || $i==46 || $i==95 || $i==126 ?
               "%c" : "%%%02x", $i)
      }
    }'
}

If your device doesn't have od, you can do everything inside the shell; this will significantly help performance (fewer calls to external program — none if printf is a builtin) and be easier to write correctly. I believe all Busybox shells support the ${VAR#PREFIX} construct to trim a prefix from a string; use it to strip the first character of the string repeatedly.

urlencode_many_printf () {
  string=$1
  while [ -n "$string" ]; do
    tail=${string#?}
    head=${string%$tail}
    case $head in
      [-._~0-9A-Za-z]) printf %c "$head";;
      *) printf %%%02x "'$head"
    esac
    string=$tail
  done
  echo
}

If printf is not a builtin but an external utility, you will again gain performance by invoking it only once for the whole function instead of once per character. Build up the format and parameters, then make a single call to printf.

urlencode_single_printf () {
  string=$1; format=; set --
  while [ -n "$string" ]; do
    tail=${string#?}
    head=${string%$tail}
    case $head in
      [-._~0-9A-Za-z]) format=$format%c; set -- "$@" "$head";;
      *) format=$format%%%02x; set -- "$@" "'$head";;
    esac
    string=$tail
  done
  printf "$format\\n" "$@"
}

This is optimal in terms of external calls (there's a single one, and you can't do it with pure shell constructs unless you're willing to enumerate all characters that need to be escaped). If most of the characters in the argument are to be passed unchanged, you can process them in a batch.

urlencode_grouped_literals () {
  string=$1; format=; set --
  while
    literal=${string%%[!-._~0-9A-Za-z]*}
    if [ -n "$literal" ]; then
      format=$format%s
      set -- "$@" "$literal"
      string=${string#$literal}
    fi
    [ -n "$string" ]
  do
    tail=${string#?}
    head=${string%$tail}
    format=$format%%%02x
    set -- "$@" "'$head"
    string=$tail
  done
  printf "$format\\n" "$@"
}

Depending on compilation options, [ (a.k.a. test) may be an external utility. We're only using it for string matching which can also be done within the shell with the case construct. Here are the last two approaches rewritten to avoid the test builtin, first going character by character:

urlencode_single_fork () {
  string=$1; format=; set --
  while case "$string" in "") false;; esac do
    tail=${string#?}
    head=${string%$tail}
    case $head in
      [-._~0-9A-Za-z]) format=$format%c; set -- "$@" "$head";;
      *) format=$format%%%02x; set -- "$@" "'$head";;
    esac
    string=$tail
  done
  printf "$format\\n" "$@"
}

and copying each literal segment in a batch:

urlencode_grouped_case () {
  string=$1; format=; set --
  while
    literal=${string%%[!-._~0-9A-Za-z]*}
    case "$literal" in
      ?*)
        format=$format%s
        set -- "$@" "$literal"
        string=${string#$literal};;
    esac
    case "$string" in
      "") false;;
    esac
  do
    tail=${string#?}
    head=${string%$tail}
    format=$format%%%02x
    set -- "$@" "'$head"
    string=$tail
  done
  printf "$format\\n" "$@"
}

I tested on my router (MIPS processor, DD-WRT-based distribution, BusyBox with ash, external printf and [). Each version is a noticeable speed improvement on the previous one. Moving to a single fork is the most significant improvement; it's the one that makes the function respond almost instantly (in human terms) as opposed to after a few seconds for a realistic long URL parameter.

share|improve this answer
    
Nice answer. I'll add that busybox (and other similar things) could replace commonly-builtin commands (printf, even echo!) with an invocation of busybox itself (I saw this behaviour in mobaXterm 3's version of busybox bash, for example) : making some script surprinsingly slow (a simple for i in ... ; do echo i ; done loop in busybox version of 'bash' will invoke busybox n times, 1 per echo, whereas in bash echo would be builtin). Try really hard to put only 1 invocation to any subcommand to avoid having multiple invocation of busybox (each with a big overhead). set -x helps to find them. –  Olivier Dulac Jan 9 '13 at 8:03
    
@OlivierDulac Whether BusyBox ash forks on builtins like echo depends on a compilation option (ENABLE_FEATURE_SH_NOFORK), which I think makes it faster but buggy in corner cases (traps, effects of special builtins). –  Gilles Jan 9 '13 at 10:58
    
thanks, good to know (I can't really recompile the one in mobaXterm, and I prefer they chose the most compatible approcah, but good to know anyway) –  Olivier Dulac Jan 9 '13 at 12:11
    
Thanks a lot! Works great! –  Serg Jan 15 '13 at 17:10
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.