# Print only when webpage updates

I'm currently developing a small auto-reply bot using Mathematica, and a key barrier is to send a reply whenever the webpage updates, in other words, new messages come in.

I cannot think of any efficient method of doing this, the only thing that came to my mind is to constantly import the HTML page by RunScheduledTaskand compare the difference between consecutive ones. But this method requires constant HTML downloading and comparing which is rather low in efficiency and low in updating speed. So I'm here to find out whether there are any better methods.

To simplify the problem, let's say, I would like Mathematica to print the present time to the front-end whenever mathematica.stackexchange.com updates.(when new questions or answers come in)

I want the update lag as short as possible and CPU consumption as little as possible. I would like to offer a bounty if a really satisfying answer(or answers) comes in~

Thanks!!!

Note: maybe package WebUnit will help. Check here for details.

Your approach is the only viable option: You have to download the website to see if it has changed. I wrote some sample code just try it, and I didn't think this solution presented a performance issue. If better performance is needed, then one should consider how to make the function compare faster in the following example, but that does not seem to be your question. My answer to your question I think is basically that there is no other way. I hope the example can be of some help nonetheless, maybe there are details in it that you hadn't considered. For example, I would not use Import, which is not asynchronous. (It could turn out that this is your problem.)

I make use of my jsoupLink package in the following example. The idea is to use something like

<< jsoupLink
html = Import["http://mathematica.stackexchange.com", "HTMLDOM"];
#["ID"] & /@ html["Select", ".question-summary"]


{"question-summary-147779", "question-summary-147755", \ "question-summary-147750", "question-summary-147778", \ "question-summary-147705", "question-summary-147699", \ "question-summary-147762", "question-summary-147772", \ "question-summary-147680", "question-summary-147763", \ "question-summary-147765", "question-summary-147714", \ "question-summary-147520", "question-summary-33652"}

To get a list of IDs of all questions on the front page of Mathematica.StackExchange.com. (It does not seem to work perfectly, but that is not important here.) I will consider the page to be updated if this list has changed.

The implementation would look something like this:

compare[new_] := Module[{html},
html = ImportString[First[new["Body"]], "HTMLDOM"];
If[
#["ID"] & /@ html["Select", ".question-summary"] == old,
Print["Same"],
old = #["ID"] & /@ html["Select", ".question-summary"];
Print["Changed"]
];
]


The following code will check once every second:

check := URLSubmit[
"http://mathematica.stackexchange.com",
HandlerFunctionsKeys -> {"Body"}
];

old = {};
`