5
$\begingroup$

Consider opening a web session such as (Mathematica 11.3)

session = StartExternalSession["WebDriver-Chrome"];
ffoxdo[cmd_] := ExternalEvaluate[session, cmd];

Then we can navigate to a webpage, e.g.

ffoxdo[ "OpenWebPage" -> "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"];

After that the respective browser is showing a website. Now I would like to import the current content of that web browser as a data table into Mathematica. If I try

Import[session, "Data"]

the operation fails, which is not too surprising. I assume there is a special syntax for that? How should I do this properly?

PS:

The actual website I would like to get data from takes a second to load the relevant data into view, which is why I am trying to access it through a session instead of using Import[httplink] (the data is missing if I do that).

EDIT:

While another question mentions a function that might lead to a solution "WebUnit`GetPageHtml", it does not solve the issue raised here per se. Making use of two more lines, we can load the html as

session = StartExternalSession["WebDriver-Chrome"];
ffoxdo[cmd_] := ExternalEvaluate[session, cmd];
iws = ExternalEvaluateWebDriver`Private`websession[];
ffoxdo[ "OpenWebPage" -> "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"];
html=WebUnit`GetPageHtml[iws];
DeleteObject[session];

However, the goal is to extract data from the html code, similarly to what the output of

Import["https://weather.com/weather/tenday/l/New+York+NY+10017:4:US","Data"]

would give. Trying Import[html,"Data"] leads to an error. How should I be doing this properly?

$\endgroup$
  • 1
    $\begingroup$ Look at WebUnit`GetPageHtml in this Q&A If that link does not answer your question, do let us know, otherwise probably you may not get an answer. $\endgroup$ – rhermans Jul 23 at 14:19
  • $\begingroup$ @rhermans The link you point to seems to suggest a useful function, but there are still some issues left. Please, see my edit above. $\endgroup$ – Kagaratsch Jul 23 at 14:36
  • $\begingroup$ Can you say what is the actual website and data you want? $\endgroup$ – rhermans Jul 23 at 15:19
  • $\begingroup$ @rhermans I am actually looking at barchart.com/stocks/quotes/SPY/options?moneyness=allRows , the data being in the table that is loaded. $\endgroup$ – Kagaratsch Jul 23 at 15:21
4
$\begingroup$

This will work on Mathematica 11.3, and not in 12 as websession[] seems to have changed.

Module[
 {
  session = StartExternalSession["WebDriver-Chrome"],
  iws, chromedo, img, links
 }, 
 chromedo[cmd_] := ExternalEvaluate[session, cmd];
 Pause[1];
 iws = ExternalEvaluateWebDriver`Private`websession[];
 Pause[1];(*Time to load chrome*)
 chromedo[
  "OpenWebPage" -> 
  "https://www.barchart.com/stocks/quotes/SPY/options?moneyness=allRows"
   ];
 Pause[15];(*Time to load the page*)
 Echo@WebUnit`GetURL[iws];
 html = WebUnit`GetPageHtml[iws];
 DeleteObject[session];
 ]
TableForm[ImportString[html, {"HTML", "Data"}][[1, 2, 2, 2, 1, 2]]]

enter image description here

$\endgroup$
6
$\begingroup$

In 11.3 you can do:

ExternalEvaluate[session, "JavascriptExecute" -> 
  "return document.documentElement.outerHTML;"]

In 12.0 and up, the syntax has change a little bit:

session = StartWebSession[]; 

WebExecute[session, "OpenWebPage" -> 
  "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"]

html = WebExecute[ session, "JavascriptExecute" -> 
  "return document.documentElement.outerHTML;"]

Note that the Wolfram Language comes with a WeatherData function built-in, so you don't need to scrape data from a web page.

Also the National Weather Service has an public API, which might give you a more structured way to get this sort of data.

You import the html above with this:

ImportString[ html, "XMLObject" ]

This gives you a Wolfram Language expression that you can traverse with Part, Take and use with functions like Cases.

If your actual interest is stocks, then you should probably be aware of the FinancialData function.

$\endgroup$
3
$\begingroup$

Here is a fairly stupid workaround to solve the above problem.

As suggested by rhermans, we can first obtain the html text of the webpage after it has finished loading:

session = StartExternalSession["WebDriver-Chrome"];
ffoxdo[cmd_] := ExternalEvaluate[session, cmd];
iws = ExternalEvaluateWebDriver`Private`websession[];
ffoxdo[ "OpenWebPage" -> "https://weather.com/weather/tenday/l/New+York+NY+10017:4:US"];
Pause[3];
html=WebUnit`GetPageHtml[iws];
DeleteObject[session];

Then, since Import cannot be used directly on a string of html text, we save it to disk and load from there:

Export["my.txt", html];
RenameFile["my.txt", "my.html"];
data = Import["my.html", "Data"];
DeleteFile["my.html"];

Now data indeed contains the output I was hoping for. But the workaround of writing to disk first is kind of unsatisfactory.

$\endgroup$
  • $\begingroup$ "since Import cannot be used directly on a string of html text".. may be ImportString? $\endgroup$ – rhermans Jul 23 at 15:12
  • $\begingroup$ @rhermans trying ImportString[html,"Data"], unfortunately the output is completely different and still contains a lot of HTML code. $\endgroup$ – Kagaratsch Jul 23 at 15:17
  • 1
    $\begingroup$ @Kagaratsch if the output is completely different that's a bug. You should report it with code to reproduce the issue. $\endgroup$ – b3m2a1 Jul 23 at 16:39
  • 2
    $\begingroup$ You need ImportString[html, {"HTML", "Data"}] $\endgroup$ – Aaron Enright Jul 25 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.