8

I have two chunks of base64 data in a bash variable. The usual line breaks within the base64 data have been replaced by spaces and the variable is basically one very long one-line string.

I can decode the two chunks of base64 data contained in the variable but I experienced some nuances when trying to do it. I'd like to understand if I am approaching this correctly or is there a better way to decode base64 data that does not contain line breaks. Here is what I have:

The first chunk is 350 characters and I can decode it successfully like this:

echo ${DATA::350} | openssl base64 -d | wc -c
256

The second chunk is 5745 characters but the above command doesn't produce the expected results. i.e:

$ echo {DATA:350} | openssl base64 -d | wc -c
432

However, it works if I put the line breaks back:

$ echo ${DATA:350} | tr ' ' "\n" | openssl base64 -d | wc -c
4240

I expect there is some line length issue that the first chunk is small enough to avoid, and it would appear to be a feature of the base64 decoder being used (the two usual ones, base64 and openssl base64, behave differently).

The base64 decoder (instead of openssl base64) stops at the first invalid character (the whitespace) and therefore just decodes the first "line" (48 bytes of output data) whereas OpenSSL outputs 432 characters (9 "lines"). The base64 command has an option to ignore garbage, so this works:

$ echo ${DATA:350} | base64 -d -i | wc -c
4240

The OpenSSL decoder doesn't appear to have such an option.

Also, removing the whitespace entirely works for base64 but not openssl base64:

$ echo ${DATA:350} | tr -d ' ' | openssl base64 -d | wc -c
400

$ echo ${DATA:350} | tr -d ' ' | base64 -d | wc -c
4240

So, in the end, I replaced the newlines and used the OpenSSL decoder because I needed to further process the decoded data anyway:

$ openssl enc -d -a -in <(echo ${DATA:350} | /usr/bin/tr ' ' "\n") -aes-256-cbc -pass file:<(echo $skey) | ...

But I'd like to understand Can OpenSSL decode base64 data that does not contain line breaks ?

  • FWIW bash can change or delete chars itself, without tr, using ${var//old[/new} -- but not at the same time as substringing. – dave_thompson_085 Aug 3 '17 at 15:43
15

If you don't need spaces then openssl will handle this with the -A option:

So:

$ ls -l sp2.bmp
-rw-r--r-- 1 sweh sweh 3000054 Apr 21 20:13 sp2.bmp
$ x=$(openssl base64 -A < sp2.bmp)                
$ echo "$x" | wc
      1       1 4000073
$ echo "$x" | openssl base64 -d -A > res
$ ls -l res
-rw-r--r-- 1 sweh sweh 3000054 Jul 30 10:00 res
$ cmp res sp2.bmp 
$ 

We can see the base64 data is all on one line, and can be decoded.

man enc explains the -A option.

If you need to keep the spaces then you'll need to remove them (either by converting to '\n' or by deleting and using -A).

  • The -A option was one of the first things I tried, but I did so before trying to remove the spaces and then "moved on" when it didn't work! The man page doesn't mention anything about spaces being a problem. Anyway, i've just tried it again with the spaces removed and it does work. It's still nasty but at least it's just a tr -d ' '. It's a shame it can't process it and ignore the whitespace (like base64 can). – starfry Jul 30 '16 at 14:13
  • 2
    RFC 4648 says that strictly speaking spaces (section 3.3) and linefeeds (section 3.1) are forbidden in base64. Section 3.3 also says implementations MUST reject data in this case. So openssl base64 -A is closer to being a strict interpretation, which you'd expect from a security encryption tool. I guess the coreutils base64 is more lenient. – Stephen Harris Jul 30 '16 at 14:37
  • 1
    OpenSSL implemented base64 in the 1990s (before 4648 or even 3548) primarily for reading and writing 'PEM' (really PEM-like) and S/MIME files, both of which require linebreaks. -A is basically "we've got the EVP_{En,De}codeBlock factors, might as well let people use them". – dave_thompson_085 Aug 3 '17 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.