1

I am trying to base 64 encode the following information:


{ "expiration": "2015-12-30T12:00:00.000Z",
  "conditions": [
    {"bucket": "sigv4examplebucket"},
    ["starts-with", "$key", "user/user1/"],
    {"acl": "public-read"},
    {"success_action_redirect": "http://sigv4examplebucket.s3.amazonaws.com/successful_upload.html"},
    ["starts-with", "$Content-Type", "image/"],
    {"x-amz-meta-uuid": "14365123651274"},
    {"x-amz-server-side-encryption": "AES256"},
    ["starts-with", "$x-amz-meta-tag", ""],

    {"x-amz-credential": "AKIAIOSFODNN7EXAMPLE/20151229/us-east-1/s3/aws4_request"},
    {"x-amz-algorithm": "AWS4-HMAC-SHA256"},
    {"x-amz-date": "20151229T000000Z" }
  ]
}

I cut and pasted the above file into a file and then ran the following:

base64 data.json

and got the following:

eyAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTMwVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25z
IjogWwogICAgeyJidWNrZXQiOiAic2lndjRleGFtcGxlYnVja2V0In0sCiAgICBbInN0YXJ0cy13
aXRoIiwgIiRrZXkiLCAidXNlci91c2VyMS8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIn0s
CiAgICB7InN1Y2Nlc3NfYWN0aW9uX3JlZGlyZWN0IjogImh0dHA6Ly9zaWd2NGV4YW1wbGVidWNr
ZXQuczMuYW1hem9uYXdzLmNvbS9zdWNjZXNzZnVsX3VwbG9hZC5odG1sIn0sCiAgICBbInN0YXJ0
cy13aXRoIiwgIiRDb250ZW50LVR5cGUiLCAiaW1hZ2UvIl0sCiAgICB7IngtYW16LW1ldGEtdXVp
ZCI6ICIxNDM2NTEyMzY1MTI3NCJ9LAogICAgeyJ4LWFtei1zZXJ2ZXItc2lkZS1lbmNyeXB0aW9u
IjogIkFFUzI1NiJ9LAogICAgWyJzdGFydHMtd2l0aCIsICIkeC1hbXotbWV0YS10YWciLCAiIl0s
CgogICAgeyJ4LWFtei1jcmVkZW50aWFsIjogIkFLSUFJT1NGT0ROTjdFWEFNUExFLzIwMTUxMjI5
L3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QifSwKICAgIHsieC1hbXotYWxnb3JpdGhtIjogIkFX
UzQtSE1BQy1TSEEyNTYifSwKICAgIHsieC1hbXotZGF0ZSI6ICIyMDE1MTIyOVQwMDAwMDBaIiB9
CiAgXQp9Cg==

But the website says that the output should be:

eyAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTMwVDEyOjAwOjAwLjAwMFoiLA0KICAiY29uZGl0aW9ucyI6IFsNCiAgICB7ImJ1Y2tldCI6ICJzaWd2NGV4YW1wbGVidWNrZXQifSwNCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci91c2VyMS8iXSwNCiAgICB7ImFjbCI6ICJwdWJsaWMtcmVhZCJ9LA0KICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL3NpZ3Y0ZXhhbXBsZWJ1Y2tldC5zMy5hbWF6b25hd3MuY29tL3N1Y2Nlc3NmdWxfdXBsb2FkLmh0bWwifSwNCiAgICBbInN0YXJ0cy13aXRoIiwgIiRDb250ZW50LVR5cGUiLCAiaW1hZ2UvIl0sDQogICAgeyJ4LWFtei1tZXRhLXV1aWQiOiAiMTQzNjUxMjM2NTEyNzQifSwNCiAgICB7IngtYW16LXNlcnZlci1zaWRlLWVuY3J5cHRpb24iOiAiQUVTMjU2In0sDQogICAgWyJzdGFydHMtd2l0aCIsICIkeC1hbXotbWV0YS10YWciLCAiIl0sDQoNCiAgICB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQUlPU0ZPRE5ON0VYQU1QTEUvMjAxNTEyMjkvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LA0KICAgIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwNCiAgICB7IngtYW16LWRhdGUiOiAiMjAxNTEyMjlUMDAwMDAwWiIgfQ0KICBdDQp9

The website does say this:

When you copy/paste the preceding policy, it should have carriage returns and new lines for your computed hash to match this value (ie. ASCII text, with CRLF line terminators).

What I did was to just open up vi and paste the JSON in insert mode. Does that introduce hidden characters in anyway or am I just doing this wrong?

1
  • Is it LF or CRLF terminated?
    – Panki
    Jul 16 '20 at 13:09
2

What I did was to just open up vi and paste the JSON in insert mode. Does that introduce hidden characters in anyway or am I just doing this wrong?

Exactly the opposite. Linux default is to use only a "line feed" byte. The website says you must have "carriage return" followed by "line feed" (CRLF). So you are not adding hidden bytes... you are failing to add hidden bytes.

Further, the base64 string they've provided shows there's no line terminator at the end of the last line. Vim will automatically add one. So instead of using vim I recommend you just decode the base64 string they provide:

echo eyAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTMwVDEyOjAwOjAwLjAwMFoiLA0KICAiY29uZGl0aW9ucyI6IFsNCiAgICB7ImJ1Y2tldCI6ICJzaWd2NGV4YW1wbGVidWNrZXQifSwNCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci91c2VyMS8iXSwNCiAgICB7ImFjbCI6ICJwdWJsaWMtcmVhZCJ9LA0KICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL3NpZ3Y0ZXhhbXBsZWJ1Y2tldC5zMy5hbWF6b25hd3MuY29tL3N1Y2Nlc3NmdWxfdXBsb2FkLmh0bWwifSwNCiAgICBbInN0YXJ0cy13aXRoIiwgIiRDb250ZW50LVR5cGUiLCAiaW1hZ2UvIl0sDQogICAgeyJ4LWFtei1tZXRhLXV1aWQiOiAiMTQzNjUxMjM2NTEyNzQifSwNCiAgICB7IngtYW16LXNlcnZlci1zaWRlLWVuY3J5cHRpb24iOiAiQUVTMjU2In0sDQogICAgWyJzdGFydHMtd2l0aCIsICIkeC1hbXotbWV0YS10YWciLCAiIl0sDQoNCiAgICB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQUlPU0ZPRE5ON0VYQU1QTEUvMjAxNTEyMjkvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LA0KICAgIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwNCiAgICB7IngtYW16LWRhdGUiOiAiMjAxNTEyMjlUMDAwMDAwWiIgfQ0KICBdDQp9 | base64 -d > data.json

Reading the AWS page you've referenced, it doesn't actually matter that your base64 encoded string does not match theirs. They have only given this as an example.

That is, they do not require your own constructed JSON objects use CRLF in the text rather than LF. As long as you sign your base64 encoded string and send that it should be fine.


If you really want to get to a file that exactly matches the AWS example by hand then you can use unix2dos (from the dos2unix package to swap line encoding) then strip the last CRLF with head.

unix2dos < data.json | head -c -2 > data2.json
5
  • I can decode their string but what I need to do is to base 64 encode an arbitrary json payload. This was a test to determine that I have the correct way to do that. I'm just seeing if I can actually reproduce their encoded string but I can't.
    – M.K.
    Jul 16 '20 at 13:27
  • One thing I noticed is that when I open the file where I pasted the JSON I see -UU at the bottom emacs bar. But after decoding the above as you suggested and reproducing the file, and opening that file I see this at the bottom: -UU-(DOS) - so my file has a different encoding.
    – M.K.
    Jul 16 '20 at 13:29
  • @M.K. and yes. LF is standard linux / unix. CRLF is standard dos / windows. CR is standard mac. It's a stupid mess that the software community never managed to fix. Jul 16 '20 at 13:50
  • OK, I'm nearly there. This is what I have done now: I copied the JSON from the website. Pasted it into vi and then used the unix2dos tool to convert the file to DOS encoding. I then base64 encoded it. I get almost exactly the same encoding now except for the last few characters in the encoding: I get BdDQp9DQo= whereas the website has BdDQp9
    – M.K.
    Jul 16 '20 at 14:01
  • Re-reading your post I figured it out. The only difference between my pasted JSON and reverse engineering their b64 encoded value to produce their JSON file is that their file has a ^M after the last curly bracket in the JSON which I don't. The unix2dos utility didn't add it.
    – M.K.
    Jul 16 '20 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.