# How can I monitor and detect when a website has changed?

I'd like to detect when a website has changed, in particular when new products are announced on a site that doesn't change to often. Here's the code I wrote but I don't think it is perfect. Also the site might block me if I ping it too many times...

content = Hash@URLFetch["www.wolfram.com"];
$c = 0; Dynamic[$c]

RunScheduledTask[$c++; If[content =!= Hash@URLFetch["www.wolfram.com"], SendMail["To" -> "me@me.com", "Subject" -> "The wolfram site has been updated!", "Body" -> {"Check wolfram now"}, "From" -> "email", "Server" -> "smtp.gmail.com", "UserName" -> "email", "Password" -> "pass", "PortNumber" -> 587, "EncryptionProtocol" -> "StartTLS"]; content = Hash@URLFetch["www.wolfram.com"]], 100]  I'd also like to create a standalone executable for running this process in the background without mathematica being open. - You can you use Hash instead of comparing the contents. – Spawn1701D Apr 16 '13 at 17:33 Good point! @Spawn1701D – user4364 Apr 16 '13 at 17:41 The most likely reason for getting blocked is because your headers will show you're not using a regular user-agent. You might want to change the headers, which you can do with URLFetch, so it looks like you're using a web browser. – Pickett Apr 16 '13 at 18:23 ## 1 Answer Instead of fetching the entire page you could fetch only the headers and track/compare the Last-Modified header. modified = URLFetch["www.wolfram.com","Headers"]["Last-Modified"];$c = 0; Dynamic[$c] RunScheduledTask[$c++;
SendMail[...];


Sorry if my syntax is a bit off, I'm a bit rusty.

PS: You would also need to check for that header being omitted, and fall back on another method.

-
This won't work. For many sites, like apple, they update every second, even though the content doesn't change. – user4364 Apr 16 '13 at 18:11
@user4364 apple.com and wolfram.com don't provide a last-modified header, and I updated my answer to mention needing a fallback option. however, if a page provides that header then it should be authoritative. if it's wrong then the page/site is broken. the explicit purpose of that header is for caching and other uses similar to what you're trying to do. – Sparr Apr 16 '13 at 19:14