8
$\begingroup$

I have this code

Hash["a","SHA"]
Hash[ToCharacterCode["a"],"SHA"]

and I was expecting to get the same result, but somehow it is different.

My question is: how does Mathematica calculate hashing for strings? From common sense it should convert a string to integer and calculate it for some integer. If so, how does it convert the string to integer?

$\endgroup$
12
$\begingroup$

As of version 11.3.0, the documentation (and implementation) of Hash have been updated.

When given a string, the hash algorithm works on the bytes of that string's UTF-8 representation.

For an example, below hashing the string "a" is the same as hashing the single byte 97 (decimal):

$Version

(* "11.3.0 for Mac OS X x86 (64-bit) (March 5, 2018)" *)

Hash["a", "SHA"] === Hash[StringToByteArray["a"], "SHA"] === Hash[ByteArray[{97}], "SHA"]

(* True *)

It is also worth noting that unlike previous versions, ByteArray does have a special meaning and the hashing is performed directly on the underlying bytes.

That applies for all hash types except the one argument form Hash[e] which can also be specified as Hash[e, "Expression"].

Addendum

Per @Kuba's request, I am adding some code that allows reproducing the results of string hashing as done in previous releases (back to version 9 anyway).

That is, of course, nothing like the actual implementation, just a mock-up intended to give the same hash values.

It has no error checking to speak of and probably is slower than it needs to be.

Clear[byteHash, stringHash];

byteHash[bytes_List, method_: "MD5"] := If[$VersionNumber >= 11.3
  ,
  Hash[ByteArray[bytes], method]
  ,
  Module[{tmp = CreateTemporary[], hash},
   BinaryWrite[tmp, bytes];
   Close[tmp];
   hash = FileHash[tmp, method];
   DeleteFile[tmp];
   hash]
]

stringHash[str_?StringQ, method_: "MD5"] := 
 Module[{bytes, cc = ToCharacterCode[str]},
  bytes = Which[
    $VersionNumber >= 11.3, 
       ToCharacterCode[str, "UTF-8"],
    $VersionNumber >= 11.2 && Max[cc] >= 512, 
       ToCharacterCode[ToString[str, CharacterEncoding -> "ISO8859-1"]],
    $VersionNumber >= 9.0, 
       Mod[cc, 256],
    True, 
       $Failed];
  byteHash[bytes, method]
]
$\endgroup$
  • $\begingroup$ Thanks for both answers. can you add how was it working for pre V11.3? $\endgroup$ – Kuba Mar 11 '18 at 5:57
  • $\begingroup$ @Kuba Maybe you should undelete your answer, it sounds right to me. The string needs to be rewritten in ISO8859-1, so that for example the single character \[Alpha] would become 8 characters: \, [, A etc. $\endgroup$ – ilian Mar 11 '18 at 6:21
  • $\begingroup$ I will try to come back and review it later, I remember I deleted it because I was getting inconsistent results on Cloud vs Windows. I think I suspected that is only a matter of picking correct encoding here and there but somehow stopped investigating. $\endgroup$ – Kuba Mar 11 '18 at 6:25
  • 1
    $\begingroup$ It seems that Hash for ByteArray was overloaded too, otherwise I'd get the same result for Hash[ByteArray[{195, 153}], "SHA256"] in V11.2 and V11.3, right? It is not the case. And due to that internal issuess and a bug you mentioned I think I will not undelete my answer. It would help a lot if you could find some time to elaborate on history of Hash for strings/byte arrays etc, many people need to support their packages for older versions too. $\endgroup$ – Kuba Mar 12 '18 at 8:37
  • 1
    $\begingroup$ @Kuba Right, it's what I meant by "unlike previous versions, ByteArray ..." $\endgroup$ – ilian Mar 12 '18 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.