I apologise if this is not the correct community.

I have a string that I believe was encrypted using an RC2 cipher. I know the secret key and the IV but I am struggling to decrypt it using OpenSSL. I know what the plain text should be.

$ echo MY_CIPHER_TEXT | openssl enc -d -base64 -rc2 -iv MY_IV

I am prompted for the decryption password, which I enter, but i always received a response

bad magic number

I believe this means openssl does not recognise MY_CIPHER_TEXT as ciphered text, but I am struggling to understand why.

Can someone help explain why i am getting the "Bad Magic Number" response?

MY_CIPHER_TEXT = nKZQD6RKk9ozeGV5WOMVL9TDZTgg9mOZjDpBDqIocR8OGC+WcB4xAwDx7XTaJNv9v+Y3sEzNphtET6sXxBd0e/0Oh6g2d0LrKls2BFHGbaMynEVW2xy4xLP40se55zdawVLGImSxgiBtf9unfIJYN4EpdPlMiiB2TuvyEoUUtqQ=

MY_VI = jqn76XOl4To=
  • security.stackexchange.com might be more suitable. – Julie Pelletier Dec 24 '16 at 7:31
  • 1
    @JuliePelletier No, it wouldn't. Using tools that are only vaguely associated with security, such as “how do I decrypt this data” or “why doesn't my ssh connect” or “how do I configure my firewall”, is off-topic there. Information Security is primarily concerned with matching security requirements with security policies; tools are only on-topic when the question is primarily about the security policy that they are enforcing or about protecting their assets. And Cryptography would also reject this question because it's about the workings of crypto, not about debugging code/tools usage. – Gilles 'SO- stop being evil' Dec 24 '16 at 23:54

Knowing the algorithm RC2 isn't enough; you also need to match the mode of operation and for some modes padding scheme. OpenSSL commandline (and for the most part the EVP API as well) defaults to CBC mode and 'PKCS5' (technically PKCS7) padding, which may or may not be correct.

openssl enc by default does password-based encryption and decryption, which means the actual key and IV (except for ECB, which has no IV) used for the cipher are derived by a hashing process called Password-Based Key Derivation Function (PBKDF) -- and a nonstandard one to boot; any argument you give as -iv is ignored -- which is good because the argument you gave is invalid anyway, see below. The OpenSSL PBKDF (like other better ones) uses a random 'salt' which must be stored in an OpenSSL-specific format at the beginning of the ciphertext, and the lack of that salt is causing your error message bad magic number. For more details see https://crypto.stackexchange.com/questions/3298/is-there-a-standard-for-openssl-interoperable-aes-encryption/35614#35614 .

Since you have the key, NOT a password, and the IV, convert them both to hex (not base64) and use:

 openssl enc -base64 -d -rc2[-mode] -K $key_in_hex -iv $iv_in_hex
 # note that's -K uppercase not -k lowercase 
 # you can use -a as a synonym for -base64 
 # For a block mode like CBC if standard PKCS5/7 padding wasn't used
 # add -nopad and handle the last few bytes manually as needed.
 # If your input is more than 76 chars per line (as your Q showed) 
 # and OpenSSL version before 1.1.0 you also need -A (uppercase).

There are many ways to convert base64 to hex, but a convenient one is:

 somevar=$( echo some_base64 | openssl base64 -d | xxd -p )
 # xxd -p outputs only the hex with no labels or ASCII etc
 # and thus is suitable as an argument to openssl enc 
 # without any processing by tools like sed, tr, awk 
  • The Hex values for key and iv solved my issues. I read the openssl man pages but missed the fact that the key and iv had to be presented in hex. So thanks for that. Unfortunately the string did not decrypt into something I was expecting so my initial premise must be wrong. I was expecting an SHA1 hash. – Michael Dec 26 '16 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.