Why does the hash of a file differ from the hash of its content?

Question

FileHash doesn't allow arguments like

FileHash[file,{"CRC32","MD5"}];

A solution could be:

fileHash[file_, hashFormatList_] := FileHash[file, #] & /@ hashFormatList;

but, as far as I understand, that would force Mathematica to read the file as many times as there are elements in hashFormatList. To avoid this waste of resources I have tried:

Quiet[Remove[fileHash]];
fileHash[file_,hashFormatList_]:=Block[
{return,fileContent},
    fileContent=Import[file,"Binary"];
    (* or fileContent=BinaryReadList[file]; *)
    return=Hash[fileContent,#]&/@hashFormatList;
Return[return];
];

but, surprisingly to me, the MD5 hash of a file differs from the MD5 hash of that file's content. What is the source of this inconsistency?

An answer to the problem of reading the file over and over again would be greatly appreciated also.

Answer 1

You need to import as "String", not "Binary". The latter returns a list of numbers, which is not exactly the same as the file contents (though there's a one-to-one mapping between the two) ... Think e.g. about the fact that integers in Mathematica can be arbitrarily large while byte values are between 0..255.

"String" will return the file contents byte for byte, and Hashing the result give the same as FileHash, provided that the same type of hashing method is selected.

Note that the default hashing method doesn't seem to be the same for Hash and FileHash.

---

Regarding avoiding reading files multiple times:

You can Import a file as "String", then treat the string as a "file" using one of several ways:

• Import form it using ImportString

• Convert it to a stream using StringToStream.

Answer 2

I believe your general concern is largely unfounded as file caching is already used.

Compare these timings performed in Mathematica 10.0.2:

Quit[]  (* fresh Kernel session *)

FileHash["file.mx", #] & /@ {"MD5"} // AbsoluteTiming

{0.422024, {258141117636363265815580588062419660198}}

FileHash["file.mx", "MD5"] // AbsoluteTiming

{0.090005, 258141117636363265815580588062419660198}

FileHash["file.mx", "CRC32"] // AbsoluteTiming

{0.114007, 3584556488}

Note that only the first call is slowed by file import.