I am trying to base 64 encode the following information:

{ "expiration": "2015-12-30T12:00:00.000Z",
  "conditions": [
    {"bucket": "sigv4examplebucket"},
    ["starts-with", "$key", "user/user1/"],
    {"acl": "public-read"},
    {"success_action_redirect": "http://sigv4examplebucket.s3.amazonaws.com/successful_upload.html"},
    ["starts-with", "$Content-Type", "image/"],
    {"x-amz-meta-uuid": "14365123651274"},
    {"x-amz-server-side-encryption": "AES256"},
    ["starts-with", "$x-amz-meta-tag", ""],

    {"x-amz-credential": "AKIAIOSFODNN7EXAMPLE/20151229/us-east-1/s3/aws4_request"},
    {"x-amz-algorithm": "AWS4-HMAC-SHA256"},
    {"x-amz-date": "20151229T000000Z" }

I cut and pasted the above file into a file and then ran the following:

base64 data.json

and got the following:


But the website says that the output should be:


The website does say this:

When you copy/paste the preceding policy, it should have carriage returns and new lines for your computed hash to match this value (ie. ASCII text, with CRLF line terminators).

What I did was to just open up vi and paste the JSON in insert mode. Does that introduce hidden characters in anyway or am I just doing this wrong?

  • Is it LF or CRLF terminated?
    – Panki
    Jul 16 '20 at 13:09

What I did was to just open up vi and paste the JSON in insert mode. Does that introduce hidden characters in anyway or am I just doing this wrong?

Exactly the opposite. Linux default is to use only a "line feed" byte. The website says you must have "carriage return" followed by "line feed" (CRLF). So you are not adding hidden bytes... you are failing to add hidden bytes.

Further, the base64 string they've provided shows there's no line terminator at the end of the last line. Vim will automatically add one. So instead of using vim I recommend you just decode the base64 string they provide:

echo eyAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTMwVDEyOjAwOjAwLjAwMFoiLA0KICAiY29uZGl0aW9ucyI6IFsNCiAgICB7ImJ1Y2tldCI6ICJzaWd2NGV4YW1wbGVidWNrZXQifSwNCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci91c2VyMS8iXSwNCiAgICB7ImFjbCI6ICJwdWJsaWMtcmVhZCJ9LA0KICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL3NpZ3Y0ZXhhbXBsZWJ1Y2tldC5zMy5hbWF6b25hd3MuY29tL3N1Y2Nlc3NmdWxfdXBsb2FkLmh0bWwifSwNCiAgICBbInN0YXJ0cy13aXRoIiwgIiRDb250ZW50LVR5cGUiLCAiaW1hZ2UvIl0sDQogICAgeyJ4LWFtei1tZXRhLXV1aWQiOiAiMTQzNjUxMjM2NTEyNzQifSwNCiAgICB7IngtYW16LXNlcnZlci1zaWRlLWVuY3J5cHRpb24iOiAiQUVTMjU2In0sDQogICAgWyJzdGFydHMtd2l0aCIsICIkeC1hbXotbWV0YS10YWciLCAiIl0sDQoNCiAgICB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQUlPU0ZPRE5ON0VYQU1QTEUvMjAxNTEyMjkvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LA0KICAgIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwNCiAgICB7IngtYW16LWRhdGUiOiAiMjAxNTEyMjlUMDAwMDAwWiIgfQ0KICBdDQp9 | base64 -d > data.json

Reading the AWS page you've referenced, it doesn't actually matter that your base64 encoded string does not match theirs. They have only given this as an example.

That is, they do not require your own constructed JSON objects use CRLF in the text rather than LF. As long as you sign your base64 encoded string and send that it should be fine.

If you really want to get to a file that exactly matches the AWS example by hand then you can use unix2dos (from the dos2unix package to swap line encoding) then strip the last CRLF with head.

unix2dos < data.json | head -c -2 > data2.json
  • I can decode their string but what I need to do is to base 64 encode an arbitrary json payload. This was a test to determine that I have the correct way to do that. I'm just seeing if I can actually reproduce their encoded string but I can't.
    – M.K.
    Jul 16 '20 at 13:27
  • One thing I noticed is that when I open the file where I pasted the JSON I see -UU at the bottom emacs bar. But after decoding the above as you suggested and reproducing the file, and opening that file I see this at the bottom: -UU-(DOS) - so my file has a different encoding.
    – M.K.
    Jul 16 '20 at 13:29
  • @M.K. and yes. LF is standard linux / unix. CRLF is standard dos / windows. CR is standard mac. It's a stupid mess that the software community never managed to fix. Jul 16 '20 at 13:50
  • OK, I'm nearly there. This is what I have done now: I copied the JSON from the website. Pasted it into vi and then used the unix2dos tool to convert the file to DOS encoding. I then base64 encoded it. I get almost exactly the same encoding now except for the last few characters in the encoding: I get BdDQp9DQo= whereas the website has BdDQp9
    – M.K.
    Jul 16 '20 at 14:01
  • Re-reading your post I figured it out. The only difference between my pasted JSON and reverse engineering their b64 encoded value to produce their JSON file is that their file has a ^M after the last curly bracket in the JSON which I don't. The unix2dos utility didn't add it.
    – M.K.
    Jul 16 '20 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.