4
$\begingroup$

There isn't much tuning freedom available in the WebExecute method. If a page takes too long to load you only see a spinner or a blank for CapturePage. Does someone see how to let the method wait a bit?

$\endgroup$
1
  • $\begingroup$ Adding a manual Pause[] in between WebExecute["OpenPage"] and WebExecute["CapturePage"] could be a possible solution, but I'm not sure if you're looking for something more elegant than that. For websites I've looked at the WebExecute["OpenPage"] command only returns once the page is loaded. Can you give an example of a website that returns but then takes addition time to load other elements of the page? $\endgroup$
    – Brady Hunt
    Apr 18, 2019 at 14:48

2 Answers 2

4
$\begingroup$

This is an unavoidable problem due to the nature of the modern web. You never know when a page has finished loading, because modern websites have Javascripts that just continuously refresh the page, so some websites will not have loaded completely after WebExecute returns from "OpenPage". When WebExecute returns, there's still a possibility that scripts change the page. The only thing you can really do, is use

WebExecute[session, "OpenPage" -> url];
Pause[n]
WebExecute[session, "CapturePage"]

to wait for a while and give the scripts on the page time to stabilize.

$\endgroup$
3
$\begingroup$

At a slight tweak on Sjoerd's solution, you can execute any JavaScript on the page to see if a particular document state has been reached (modern web pages are almost never 'static' so you have to do something more specific to ascertain you are in the state you want to capture an image of):

result = WebExecute[session, "JavaScriptExecute" -> "... javascript code..."]

And then make a decision based on what result is.

$\endgroup$
6
  • 1
    $\begingroup$ Will consider Sjoerd's proposal as the solution, since it does CapturePage without extra JavaScript. Thanks for the suggestion though. $\endgroup$ Apr 18, 2019 at 18:57
  • $\begingroup$ @Arnoud Buzing, in my experiments with WebExecute with JavascripExecute parameter, it doesn't typically return a value. Is that different for you? $\endgroup$
    – Victor K.
    Dec 8, 2021 at 9:10
  • 1
    $\begingroup$ @VictorK. Even when you use the return statement? $\endgroup$ Dec 20, 2021 at 20:57
  • $\begingroup$ @ArnoudBuzing ah, you are right, of course - I'm not sure why I didn't think about using return, but it works pretty well: e.g., WebExecute["JavascriptExecute" -> "return([1,2,'this'])"] return a list as you would expect, and WebExecute["JavascriptExecute" -> "return(document.body)"] returns a rule {"ELEMENT" -> "0.5427856833586193-1"} where the element can be feed to e.g. WebExecute("ElementText"->id). $\endgroup$
    – Victor K.
    Dec 21, 2021 at 10:35
  • $\begingroup$ @ArnoudBuzing thanks for writing WebUnit, btw - very useful package, I'm using it extensively to extract information from a bunch of dynamic web pages that has no other recourse. I'm sure you have a ton of other more important tasks, but it would be cool if WebExecute would support FindElementsFromElement (w3c.github.io/webdriver/#dfn-find-elements-from-element) eventually - it's one of the few commands defined by WebDriver that's not currently supported. Here is a use case: you want to extract a bunch of information fields (phone, email, name, etc.) from a bunch of contacts. $\endgroup$
    – Victor K.
    Dec 21, 2021 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.