OpenSSL Usage to decrypt a string

Question

I apologise if this is not the correct community.

I have a string that I believe was encrypted using an RC2 cipher. I know the secret key and the IV but I am struggling to decrypt it using OpenSSL. I know what the plain text should be.

$ echo MY_CIPHER_TEXT | openssl enc -d -base64 -rc2 -iv MY_IV

I am prompted for the decryption password, which I enter, but i always received a response

bad magic number

I believe this means openssl does not recognise MY_CIPHER_TEXT as ciphered text, but I am struggling to understand why.

Can someone help explain why i am getting the "Bad Magic Number" response?

MY_CIPHER_TEXT = nKZQD6RKk9ozeGV5WOMVL9TDZTgg9mOZjDpBDqIocR8OGC+WcB4xAwDx7XTaJNv9v+Y3sEzNphtET6sXxBd0e/0Oh6g2d0LrKls2BFHGbaMynEVW2xy4xLP40se55zdawVLGImSxgiBtf9unfIJYN4EpdPlMiiB2TuvyEoUUtqQ=

MY_VI = jqn76XOl4To=

Answer 1

Knowing the algorithm RC2 isn't enough; you also need to match the mode of operation and for some modes padding scheme. OpenSSL commandline (and for the most part the EVP API as well) defaults to CBC mode and 'PKCS5' (technically PKCS7) padding, which may or may not be correct.

openssl enc by default does password-based encryption and decryption, which means the actual key and IV (except for ECB, which has no IV) used for the cipher are derived by a hashing process called Password-Based Key Derivation Function (PBKDF) -- and a nonstandard one to boot; any argument you give as -iv is ignored -- which is good because the argument you gave is invalid anyway, see below. The OpenSSL PBKDF (like other better ones) uses a random 'salt' which must be stored in an OpenSSL-specific format at the beginning of the ciphertext, and the lack of that salt is causing your error message bad magic number. For more details see https://crypto.stackexchange.com/questions/3298/is-there-a-standard-for-openssl-interoperable-aes-encryption/35614#35614 .

Since you have the key, NOT a password, and the IV, convert them both to hex (not base64) and use:

 openssl enc -base64 -d -rc2[-mode] -K $key_in_hex -iv $iv_in_hex
 # note that's -K uppercase not -k lowercase 
 # you can use -a as a synonym for -base64 
 # For a block mode like CBC if standard PKCS5/7 padding wasn't used
 # add -nopad and handle the last few bytes manually as needed.
 # If your input is more than 76 chars per line (as your Q showed) 
 # and OpenSSL version before 1.1.0 you also need -A (uppercase).

There are many ways to convert base64 to hex, but a convenient one is:

 somevar=$( echo some_base64 | openssl base64 -d | xxd -p )
 # xxd -p outputs only the hex with no labels or ASCII etc
 # and thus is suitable as an argument to openssl enc 
 # without any processing by tools like sed, tr, awk