Import data from a current web session?

Question

Consider opening a web session such as (Mathematica 11.3)

session = StartExternalSession["WebDriver-Chrome"];
ffoxdo[cmd_] := ExternalEvaluate[session, cmd];

Then we can navigate to a webpage, e.g.

ffoxdo[ "OpenWebPage" -> "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"];

After that the respective browser is showing a website. Now I would like to import the current content of that web browser as a data table into Mathematica. If I try

Import[session, "Data"]

the operation fails, which is not too surprising. I assume there is a special syntax for that? How should I do this properly?

PS:

The actual website I would like to get data from takes a second to load the relevant data into view, which is why I am trying to access it through a session instead of using Import[httplink] (the data is missing if I do that).

EDIT:

While another question mentions a function that might lead to a solution "WebUnit`GetPageHtml", it does not solve the issue raised here per se. Making use of two more lines, we can load the html as

session = StartExternalSession["WebDriver-Chrome"];
ffoxdo[cmd_] := ExternalEvaluate[session, cmd];
iws = ExternalEvaluateWebDriver`Private`websession[];
ffoxdo[ "OpenWebPage" -> "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"];
html=WebUnit`GetPageHtml[iws];
DeleteObject[session];

However, the goal is to extract data from the html code, similarly to what the output of

Import["https://weather.com/weather/tenday/l/New+York+NY+10017:4:US","Data"]

would give. Trying Import[html,"Data"] leads to an error. How should I be doing this properly?

Answer 1

This will work on Mathematica 11.3, and not in 12 as websession[] seems to have changed.

Module[
 {
  session = StartExternalSession["WebDriver-Chrome"],
  iws, chromedo, img, links
 }, 
 chromedo[cmd_] := ExternalEvaluate[session, cmd];
 Pause[1];
 iws = ExternalEvaluateWebDriver`Private`websession[];
 Pause[1];(*Time to load chrome*)
 chromedo[
  "OpenWebPage" -> 
  "https://www.barchart.com/stocks/quotes/SPY/options?moneyness=allRows"
   ];
 Pause[15];(*Time to load the page*)
 Echo@WebUnit`GetURL[iws];
 html = WebUnit`GetPageHtml[iws];
 DeleteObject[session];
 ]
TableForm[ImportString[html, {"HTML", "Data"}][[1, 2, 2, 2, 1, 2]]]

Answer 2

In 11.3 you can do:

ExternalEvaluate[session, "JavascriptExecute" -> 
  "return document.documentElement.outerHTML;"]

In 12.0 and up, the syntax has change a little bit:

session = StartWebSession[]; 

WebExecute[session, "OpenWebPage" -> 
  "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"]

html = WebExecute[ session, "JavascriptExecute" -> 
  "return document.documentElement.outerHTML;"]

Note that the Wolfram Language comes with a WeatherData function built-in, so you don't need to scrape data from a web page.

Also the National Weather Service has an public API, which might give you a more structured way to get this sort of data.

You import the html above with this:

ImportString[ html, "XMLObject" ]

This gives you a Wolfram Language expression that you can traverse with Part, Take and use with functions like Cases.

If your actual interest is stocks, then you should probably be aware of the FinancialData function.

Answer 3

Here is a fairly stupid workaround to solve the above problem.

As suggested by rhermans, we can first obtain the html text of the webpage after it has finished loading:

session = StartExternalSession["WebDriver-Chrome"];
ffoxdo[cmd_] := ExternalEvaluate[session, cmd];
iws = ExternalEvaluateWebDriver`Private`websession[];
ffoxdo[ "OpenWebPage" -> "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"];
Pause[3];
html=WebUnit`GetPageHtml[iws];
DeleteObject[session];

Then, since Import cannot be used directly on a string of html text, we save it to disk and load from there:

Export["my.txt", html];
RenameFile["my.txt", "my.html"];
data = Import["my.html", "Data"];
DeleteFile["my.html"];

Now data indeed contains the output I was hoping for. But the workaround of writing to disk first is kind of unsatisfactory.