Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Objective: I am trying to save a HTML file using the URLFetch function (native to Mathematica 9 onward). Sometimes the file is huge.

Question: Is there a way to check/verify (using the headers or other parameters native to URLFetch) that the COMPLETE PAGE HAS BEEN DOWNLOADED, i.e. the page was not partially downloaded? Is there a protocol say like MD5 for file downloads, which allows one to confirm that the HTML file's contents were completely downloaded?

(or can I be assured that URLFetch returns/transfers control ONLY after the COMPLETE page has been fetched?)

Related questions :

  • IF the download was partial (not complete), will the "STATUS" (from the URLFetch) still be 200 ?
  • Can this be solved using Import function
share|improve this question
add comment

2 Answers

If you're downloading a website it may not be entirely static. So you need to download the file and check the headers at the same time, you cannot do it separately. Therefore URLSave is easier to use, even though URLFetch can also to be used. Like Joel Klein says there isn't a checksum by default for every page. However, we could use the header Content-Length to make sure the entire page was downloaded.

headers = URLSave["http://mathematica.stackexchange.com", "~/Desktop/test2.html", "Headers"];
headers /. {___, {"Content-Length", y_}, ___} :> y

"49933"

FileByteCount["~/Desktop/test2.html"]

49933

What's happening is that the server told us the body of the file is supposed to be a certain number of bytes. We download the file and then check the number of bytes. If the file is as big as it should be, the download was completed.

The headers will be the same regardless of whether you managed to complete the transfer or not, however I highly doubt URLFetch would return anything if it couldn't finish.

share|improve this answer
add comment

URLFetch does return only when it finishes processing the request. Knowing whether the request was successful is a different matter, it depends on where along the chain things break.

In general, you can verify the download by obtaining a checksum or hash value and then comparing it with the value you get on the downloaded data. But that's up to you and the provider of the original file to know what hash algorithm was used and what checksum/hash value to expect.

Import on a URL doesn't give you any better tools than URLFetch in this case, URLFetch has a richer, more complete set of features for making HTTP requests.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.