5
$\begingroup$

I'm currently developing a small auto-reply bot using Mathematica, and a key barrier is to send a reply whenever the webpage updates, in other words, new messages come in.

I cannot think of any efficient method of doing this, the only thing that came to my mind is to constantly import the HTML page by RunScheduledTaskand compare the difference between consecutive ones. But this method requires constant HTML downloading and comparing which is rather low in efficiency and low in updating speed. So I'm here to find out whether there are any better methods.

To simplify the problem, let's say, I would like Mathematica to print the present time to the front-end whenever mathematica.stackexchange.com updates.(when new questions or answers come in)

I want the update lag as short as possible and CPU consumption as little as possible. I would like to offer a bounty if a really satisfying answer(or answers) comes in~

Thanks!!!


Note: maybe package WebUnit will help. Check here for details.

$\endgroup$
5
$\begingroup$

Your approach is the only viable option: You have to download the website to see if it has changed. I wrote some sample code just try it, and I didn't think this solution presented a performance issue. If better performance is needed, then one should consider how to make the function compare faster in the following example, but that does not seem to be your question. My answer to your question I think is basically that there is no other way. I hope the example can be of some help nonetheless, maybe there are details in it that you hadn't considered. For example, I would not use Import, which is not asynchronous. (It could turn out that this is your problem.)

I make use of my jsoupLink package in the following example. The idea is to use something like

<< jsoupLink`
html = Import["http://mathematica.stackexchange.com", "HTMLDOM"];
#["ID"] & /@ html["Select", ".question-summary"]

{"question-summary-147779", "question-summary-147755", \ "question-summary-147750", "question-summary-147778", \ "question-summary-147705", "question-summary-147699", \ "question-summary-147762", "question-summary-147772", \ "question-summary-147680", "question-summary-147763", \ "question-summary-147765", "question-summary-147714", \ "question-summary-147520", "question-summary-33652"}

To get a list of IDs of all questions on the front page of Mathematica.StackExchange.com. (It does not seem to work perfectly, but that is not important here.) I will consider the page to be updated if this list has changed.

The implementation would look something like this:

compare[new_] := Module[{html},
  html = ImportString[First[new["Body"]], "HTMLDOM"];
  If[
   #["ID"] & /@ html["Select", ".question-summary"] == old,
   Print["Same"],
   old = #["ID"] & /@ html["Select", ".question-summary"];
   Print["Changed"]
   ];
  ]

The following code will check once every second:

check := URLSubmit[
   "http://mathematica.stackexchange.com",
   HandlerFunctions -> <|"BodyReceived" -> compare|>,
    HandlerFunctionsKeys -> {"Body"}
   ];

old = {};
RunScheduledTask[check, 1]

You have to go to Window -> Messages to see what the code is printing.

The code is downloading the website once every second, but it is not really noticeable on my computer. If it is a problem for you, please specify what kind of performance you are looking for.

$\endgroup$
  • $\begingroup$ What about directly intercept the data packages communications? Would this speed things up? This don't require any comparation of website contents. Is this possible in Mathematica? $\endgroup$ – Wjx Jun 7 '17 at 3:33
  • $\begingroup$ @Wjx How do you know if the website has been updated if you don't compare the content between the two versions? $\endgroup$ – C. E. Jun 7 '17 at 3:57
  • $\begingroup$ For example, you might have noticed that even if we don't refresh the MSE website, when new messages or contents comes in, there will be a indicator on the top of the page showing "1 new question" or so, To let the explorer know there's a new question, there must be some data packages comes in and deliver this information. Maybe I could intercept the data packages sending in, if I find some data package with contents like "there's a new question", I know it has been updated. $\endgroup$ – Wjx Jun 7 '17 at 5:35
  • $\begingroup$ @Wjx There are two ways in which Stack Exchange could achieve that. The old way is to download a file over and over again which contains information about what packages should go on the front page. The newer is to establish a socket connection to the server. Yes, these things can be reverse engineered so that you can also update your application in the same way. But you would have to reverse engineer for every website, there is no general way. And there is no way to do this with Mathematica, the Chrome developer tool or something like that is what's needed to reverse engineer their solution. $\endgroup$ – C. E. Jun 7 '17 at 11:27
  • 1
    $\begingroup$ if you don't mind, I would like to keep this question open for another few days before I choose your answer. Maybe some new thoughts could come in~ $\endgroup$ – Wjx Jun 7 '17 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.