Is it possible to verify the accuracy of a file downloaded using URLFetch

Question

Objective: I am trying to save a HTML file using the URLFetch function (native to Mathematica 9 onward). Sometimes the file is huge.

Question: Is there a way to check/verify (using the headers or other parameters native to URLFetch) that the COMPLETE PAGE HAS BEEN DOWNLOADED, i.e. the page was not partially downloaded? Is there a protocol say like MD5 for file downloads, which allows one to confirm that the HTML file's contents were completely downloaded?

(or can I be assured that URLFetch returns/transfers control ONLY after the COMPLETE page has been fetched?)

Related questions :

• IF the download was partial (not complete), will the "STATUS" (from the URLFetch) still be 200 ?
• Can this be solved using Import function

Answer 1

If you're downloading a website it may not be entirely static. So you need to download the file and check the headers at the same time, you cannot do it separately. Therefore URLSave is easier to use, even though URLFetch can also to be used. Like Joel Klein says there isn't a checksum by default for every page. However, we could use the header Content-Length to make sure the entire page was downloaded.

headers = URLSave["http://mathematica.stackexchange.com", "~/Desktop/test2.html", "Headers"];
headers /. {___, {"Content-Length", y_}, ___} :> y

"49933"

FileByteCount["~/Desktop/test2.html"]

49933

What's happening is that the server told us the body of the file is supposed to be a certain number of bytes. We download the file and then check the number of bytes. If the file is as big as it should be, the download was completed.

The headers will be the same regardless of whether you managed to complete the transfer or not, however I highly doubt URLFetch would return anything if it couldn't finish.

Answer 2

URLFetch does return only when it finishes processing the request. Knowing whether the request was successful is a different matter, it depends on where along the chain things break.

In general, you can verify the download by obtaining a checksum or hash value and then comparing it with the value you get on the downloaded data. But that's up to you and the provider of the original file to know what hash algorithm was used and what checksum/hash value to expect.

Import on a URL doesn't give you any better tools than URLFetch in this case, URLFetch has a richer, more complete set of features for making HTTP requests.