4
$\begingroup$

I'd like to detect when a website has changed, in particular when new products are announced on a site that doesn't change to often. Here's the code I wrote but I don't think it is perfect. Also the site might block me if I ping it too many times...

content = Hash@URLFetch["www.wolfram.com"];
$c = 0; Dynamic[$c]

RunScheduledTask[$c++; 
 If[content =!= Hash@URLFetch["www.wolfram.com"], 
  SendMail["To" -> "[email protected]", 
   "Subject" -> "The wolfram site has been updated!", 
   "Body" -> {"Check wolfram now"}, 
   "From" -> "email", "Server" -> "smtp.gmail.com",
    "UserName" -> "email", "Password" -> "pass",
    "PortNumber" -> 587, "EncryptionProtocol" -> "StartTLS"];
     content = Hash@URLFetch["www.wolfram.com"]], 100]

I'd also like to create a standalone executable for running this process in the background without mathematica being open.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ You can you use Hash instead of comparing the contents. $\endgroup$
    – Spawn1701D
    Apr 16, 2013 at 17:33
  • $\begingroup$ Good point! @Spawn1701D $\endgroup$
    – user4364
    Apr 16, 2013 at 17:41
  • $\begingroup$ The most likely reason for getting blocked is because your headers will show you're not using a regular user-agent. You might want to change the headers, which you can do with URLFetch, so it looks like you're using a web browser. $\endgroup$
    – C. E.
    Apr 16, 2013 at 18:23
  • $\begingroup$ I think the CloudDeploy can help.. $\endgroup$
    – yode
    Jun 12, 2017 at 15:27

1 Answer 1

Reset to default
6
$\begingroup$

Instead of fetching the entire page you could fetch only the headers and track/compare the Last-Modified header.

modified = URLFetch["www.wolfram.com","Headers"]["Last-Modified"];
$c = 0; Dynamic[$c]

RunScheduledTask[$c++; 
 If[modified =!= URLFetch["www.wolfram.com","Headers"]["Last-Modified"], 
  SendMail[...];
  modified = URLFetch["www.wolfram.com","Headers"]["Last-Modified"]], 100]

Sorry if my syntax is a bit off, I'm a bit rusty.

PS: You would also need to check for that header being omitted, and fall back on another method.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ This won't work. For many sites, like apple, they update every second, even though the content doesn't change. $\endgroup$
    – user4364
    Apr 16, 2013 at 18:11
  • 1
    $\begingroup$ @user4364 apple.com and wolfram.com don't provide a last-modified header, and I updated my answer to mention needing a fallback option. however, if a page provides that header then it should be authoritative. if it's wrong then the page/site is broken. the explicit purpose of that header is for caching and other uses similar to what you're trying to do. $\endgroup$
    – Sparr
    Apr 16, 2013 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.