Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'd like to detect when a website has changed, in particular when new products are announced on a site that doesn't change to often. Here's the code I wrote but I don't think it is perfect. Also the site might block me if I ping it too many times...

content = Hash@URLFetch["www.wolfram.com"];
$c = 0; Dynamic[$c]

RunScheduledTask[$c++; 
 If[content =!= Hash@URLFetch["www.wolfram.com"], 
  SendMail["To" -> "me@me.com", 
   "Subject" -> "The wolfram site has been updated!", 
   "Body" -> {"Check wolfram now"}, 
   "From" -> "email", "Server" -> "smtp.gmail.com",
    "UserName" -> "email", "Password" -> "pass",
    "PortNumber" -> 587, "EncryptionProtocol" -> "StartTLS"];
     content = Hash@URLFetch["www.wolfram.com"]], 100]

I'd also like to create a standalone executable for running this process in the background without mathematica being open.

share|improve this question
1  
You can you use Hash instead of comparing the contents. –  Spawn1701D Apr 16 '13 at 17:33
    
Good point! @Spawn1701D –  user4364 Apr 16 '13 at 17:41
    
The most likely reason for getting blocked is because your headers will show you're not using a regular user-agent. You might want to change the headers, which you can do with URLFetch, so it looks like you're using a web browser. –  Pickett Apr 16 '13 at 18:23
add comment

1 Answer 1

Instead of fetching the entire page you could fetch only the headers and track/compare the Last-Modified header.

modified = URLFetch["www.wolfram.com","Headers"]["Last-Modified"];
$c = 0; Dynamic[$c]

RunScheduledTask[$c++; 
 If[modified =!= URLFetch["www.wolfram.com","Headers"]["Last-Modified"], 
  SendMail[...];
  modified = URLFetch["www.wolfram.com","Headers"]["Last-Modified"]], 100]

Sorry if my syntax is a bit off, I'm a bit rusty.

PS: You would also need to check for that header being omitted, and fall back on another method.

share|improve this answer
1  
This won't work. For many sites, like apple, they update every second, even though the content doesn't change. –  user4364 Apr 16 '13 at 18:11
    
@user4364 apple.com and wolfram.com don't provide a last-modified header, and I updated my answer to mention needing a fallback option. however, if a page provides that header then it should be authoritative. if it's wrong then the page/site is broken. the explicit purpose of that header is for caching and other uses similar to what you're trying to do. –  Sparr Apr 16 '13 at 19:14
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.