HMAC correct on 11.2 but wrong and different on 11.3 running same code

Question

If you run this code on 11.2

    hmacF[method_, message_, key_] := Module[

  {keyLen, dkey, opad, ipad, blocksize},

  blocksize =
   If[method === "SHA384" || method === "SHA512",
    128,
    64
    ];

  keyLen = StringLength[key];

  dkey = If[keyLen > blocksize,
    IntegerString[Hash[key, method], 16],
    StringPadRight[key, blocksize, FromCharacterCode[0]]
    ];

  {opad, ipad} = 
   FromCharacterCode[
      BitXor[ToCharacterCode@
        StringRepeat[FromCharacterCode[FromDigits[#, 16]], blocksize],
        ToCharacterCode@dkey]] & /@ {"5c", "36"};

  IntegerString[
   Hash[
    StringJoin[opad, 
     ExportString[
      IntegerDigits[Hash[StringJoin[ipad, message], method], 256], 
      "Binary"]],
    method],
   16, blocksize]
  ]

On something we know the answer to https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md

hmacF["SHA256","symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=0.1&recvWindow=5000&timestamp=1499827319559","NhqPtmdSJYdKjVHjA7PZj4Mge3R5YNiP1e3UZjInClVN65XAbvqqM6A7H5fATj0j"]

you get the correct answer

(*c8db56825ae71d6d79447849e617115f4a920fa2acdcab2b053c4b2838bd6b71*)

Run the same code on 11.3 and you get

(*88fdd5c8c2226ff025e290b37dec36d4bc6f84affb9928950fd865f9a0fd475e*)

Why the difference?? 11.3 is not the correct HMAC result.

Answer 1

Indeed, Hash has changed for 11.3 in several ways.

The one relevant here is that strings are now hashed according to their UTF-8 representation.

As a simple example of the difference, consider the following string, which has only one character.

c = FromCharacterCode[217]; (* "\[CapitalUGrave]" *)

Its UTF-8 representation consists of the two (decimal) bytes {195, 153}, so in version 11.3 the hash would be computed from those two bytes, while in 11.2 the hash would be computed from the single (decimal) byte 217.

Then the two corresponding hash results would be different, namely

(* new hash value *)
Hash[c, "SHA256"]
Hash[StringToByteArray[c], "SHA256"] 
Hash[ByteArray[{195, 153}], "SHA256"]

(*115129295529739515318262490854329296079790445441298973994137838100166123599005*)
(* or as a hex string: *)
(* fe88df3f083e08f99f796b5ba7f6eedbe14d45241cf5e0944caf17f55ac2349d *)

which agrees with various online hash calculators, e.g. http://new.md5calc.com/hash-calc/sha256/%C3%99 and

(* old hash value *)
Developer`LegacyHash[c, "SHA256"]
Hash[StringToByteArray[c, "ISO8859-1"], "SHA256"]
Hash[ByteArray[{217}], "SHA256"]

(* 11345241613194184164858110515603144575751235911333201083896508434490695406828*)
(* or as a hex string: *)
(* 19152ddfba193b5b09fcb80d1bba5248f36027c06e81670db5a7146fb654d4ec *)

where all inputs were evaluated with 11.3.0.

The hmacF implementation uses strings in a somewhat awkward way as data containers. It is easier to take advantage of several new Hash enhancements instead, like operating on byte arrays and returning the hashes in different formats.

For example,

hmacFnew[method_, message_, key_] := 
 Module[{dkey, dmsg, opad, ipad, blocksize}, 
  blocksize = If[method === "SHA384" || method === "SHA512", 128, 64];
  {dkey, dmsg} = StringToByteArray /@ {key, message};
  If[Length[dkey] > blocksize, dkey = Hash[dkey, method, "ByteArray"]];
  dkey = Normal[dkey];
  If[Length[dkey] < blocksize, dkey = PadRight[dkey, blocksize, 0]];
  {opad, ipad} = ByteArray[BitXor[dkey, ConstantArray[#, blocksize]]]& /@ {92,54};
  Hash[Join[opad,Hash[Join[ipad,dmsg], method, "ByteArray"]], method, "HexString"]
]

hmacFnew["SHA256", \
"symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=0.\
1&recvWindow=5000&timestamp=1499827319559", \
"NhqPtmdSJYdKjVHjA7PZj4Mge3R5YNiP1e3UZjInClVN65XAbvqqM6A7H5fATj0j"]

(* c8db56825ae71d6d79447849e617115f4a920fa2acdcab2b053c4b2838bd6b71 *)

If you don't feel like modifying your code, another way to get the previous behavior is to use Developer`LegacyHash, say

Block[{Hash = Developer`LegacyHash}, 
 hmacF["SHA256", 
  "symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=\
0.1&recvWindow=5000&timestamp=1499827319559", 
  "NhqPtmdSJYdKjVHjA7PZj4Mge3R5YNiP1e3UZjInClVN65XAbvqqM6A7H5fATj0j"]]

(* c8db56825ae71d6d79447849e617115f4a920fa2acdcab2b053c4b2838bd6b71 *)