-hmac takes the key as an argument (see manual), so your command asks for an HMAC using the key
hexkey:... is taken as a filename, since it doesn't start with a dash, and
openssl doesn't take options after filenames, so the following
-out is also a filename.
To get the HMAC with a key given as a hex string, you'll need to use
-mac hmac and
-macopt hexkey:<key>. Note that using
-hmac <key> and
-mac hmac together doesn't work, and
openssl dgst -sha256 -hmac abc <<< "message"
openssl dgst -sha256 -hmac abc -macopt hexkey:12345678 <<< "message"
openssl dgst -sha256 -mac hmac -macopt hexkey:616263 <<< "message"
perl -MDigest::HMAC=hmac_hex -MDigest::SHA=sha256 \
-le 'print(hmac_hex("message\n", "abc", \&sha256))'
All give the hash
99592e56fcde028fb41882668b0cbfa0119116f9cf111d285f5cedb000cfc45a which agrees with a random online HMAC calculator for message
616263 in hex. (Note the newline at the end of
So, it seems you'd probably want
openssl dgst -sha256 -mac hmac -macopt hexkey:$(cat mykey.txt) -out hmac.txt /bin/ps
Since we're talking about cryptography, which is hard; and OpenSSL, which doesn't always have the most easy-to-use interfaces, I would suggest also verifying everything yourself, at least twice, instead of taking my word for it.