2

I'm using openssl dgst -sha1 -binary to get hash values of my strings in binary format.

(I'm using -binary flag because my version of openssl adds "stdout" before each hash value on default output, and -binary helps to avoid it, therefore it's easier to store hash results in binary format for further processing (so I can just use xxd -p when I want hex values instead of manually cutting out that "stdout" from each string)

So, the binary output of openssl dgst -sha1 -binary for a "Hello!" string in Cygwin console will look like this: _▒▒"q▒%▒a▒▒▒▒.&C▒0N▒Q▒▒vH&8i

Now I create a new variable with this result and concatenate it with another variable, which value is not in binary format (i.e. "World"). So my new variable now looks like

_▒▒"q▒%▒a▒▒▒▒.&C▒0N▒Q▒▒vH&8iWorld

Then I generate another hash for this concatenated string and compare it to the one I got using default Java hashing libraries (MessageDigest). On this step, however, hashes obtained via shell and Java don't match (and I need to get exactly the same value as the one generated on Java side).

So I suppose that my "World" string should be in binary format as well to match my Java hash output (because as long as I generate hashes for concatenated binary values all hashes match). However, I don't know how to convert my "World" string to binary format in shell. Any ideas?

  • To be clear: You are generating a hash, adding a salt and then hashing the whole thing again? O.o – muru Jan 19 '15 at 12:20
  • @muru Yes. And there's no way I can change it (although I'd love to), I can only adapt my shell script to match values from existing Java code. – C_U Jan 19 '15 at 12:34
  • I'm sure there's a better way to do this, even on Cygwin... What's the final hex output produced by the Java code when you feed it "Hello!" and "World"? – PM 2Ring Jan 19 '15 at 12:40
5

You can't store binary data (binary data generally refers to data with arbitrary byte values, not only byte values that form valid characters but is not special otherwise) in bash variables as bash doesn't support storing the 0 byte value in its variables (and remember you can't pass such strings in arguments to commands as those are NUL delimited strings).

You can in zsh though. Also remember that command substitution strips trailing newline characters (0xa bytes, maybe different on Cygwin), so it's probably better to use read here:

$ echo 323 | openssl dgst -sha1 -binary | hd
00000000  3a 8b 03 4a 5d 00 e9 07  b2 9e 0a 61 b3 54 db 45  |:..J]......a.T.E|
00000010  63 4b 37 b0                                       |cK7.|
00000014

See how that contains both a 0 byte and newline character (0xa)

$ echo 323 | openssl dgst -sha1 -binary | IFS= LC_ALL=C read -ru0 -k20 var &&
  var=${var}World
$ printf %s $var | hd
00000000  3a 8b 03 4a 5d 00 e9 07  b2 9e 0a 61 b3 54 db 45  |:..J]......a.T.E|
00000010  63 4b 37 b0 57 6f 72 6c  64                       |cK7.World|
00000019

Note again that you can only pass that variable to builtin commands (printf...).

Now, if all you want is hash it again, then it's just

(echo 323 | openssl dgst -sha1 -binary; printf %s World) |
  openssl dgst -sha1 -binary

no need for a variable.

  • Thank you very much for your detailed explanation! However, I tried using (printf %s World | echo -n Hello openssl dgst -sha1 -binary; ) | openssl dgst -sha1 -binary | xxd -p And the value still didn't match. I also tried echo -n to avoid empty lines and I added xxd -p for output in hex. My code is slightly different from yours because in fact I want the result to look like SHA1(World+SHA1(Hello)), not vice versa. Did I change your code correctly? And I see that you are using both printf %s and echo, there must be a reason for that, right? What is the difference? – C_U Jan 19 '15 at 14:26
  • @C_U Then: (printf %s World; printf %s Hello | sha1) | sha1. piping things to echo doesn't make sense. echo doesn't read anything, it writes its argument (after transformation which is one of the reasons why you should use printf instead). – Stéphane Chazelas Jan 19 '15 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.