2
$\begingroup$

I have been using WebSession and WebExecute to pull in information from a website. Using LocateElements withCSSSelector has allowed me to easily identify sections on a website and pull in the text. However, I am struggling to figure out a way to extract the underlying hyperlink from a linked image. I can identify the WebElementObject but I am at a loss for how to get the link from it. Perhaps it isn't possible with the tools I am using, or perhaps I am overlooking something simple. Perhaps I could use a JavaScriptExecute argument?

My solution right now is to pull in all the page hyperlinks and use several other functions to figure out which is the appropriate link.

Here is the code to pull in the text from the elements. In this case, I am looking at the links in the footer of the wolfram website. What I want is the hyperlinks behind that text.

session = StartWebSession["Chrome"];
WebExecute[session, "OpenPage" -> "www.wolfram.com"];
WebExecute[session, 
 "ElementText" -> 
  WebExecute[session, 
   "LocateElements" -> "CSSSelector" -> "#_footer-bc > ul > li > a"]]

{"Legal", "Privacy Policy", "Site Map", "WolframAlpha.com", "WolframCloud.com"}

It would be great to pull any of the attributes from a WebElement that you have already identified as that could come in handy in a lot of ways.

$\endgroup$
  • $\begingroup$ It would help if you would provide some example code and and a sample html in which you would specify what part of what element you're looking for. Otherwise this is too abstract... $\endgroup$ – Shwouchk Jan 25 at 2:42
  • $\begingroup$ Just added some example code on identifying elements and then pulling text. $\endgroup$ – kickert Jan 25 at 14:37
2
+50
$\begingroup$

As stated in the comment, I think the question was not very well specified so it would be quite hard to give a good satisfactory answer. However, I did not know about WebExecute before and got curious about it. After playing around a bit and reading some docs, my conclusion is that there's no direct way to get the HTML or to manipulate the document using the WebElementObject objects returned by WebExecute.

However, a workaround presents itself in the form of the JavascriptExecute command. The idea is to use a javascript snippet to extract the sought after element(s) and then to add the entire html of said elements as the text of a new element with a given unique ID. Then we return to Mathematica where we find this new element and use ElementText to extract the html of the elements we extracted in JS. Since no HTML was provided I demonstrate on an arbitrary page and assume that OP would be able to narrow to the required element with javascript.

Here is sample code to extract an img tag from https://imgur.com/gallery/uGNoByn:

First, here is a snippet of the original HTML found when inspecting the main image on that page:

<div class="image post-image"><img alt="" src="//i.imgur.com/qE3kW4Z.jpg" style="max-width: 100%; min-height: 574px;" original-title=""></div>

For the sake of this example we'll use the image class to find the appropriate elements and extract their html:

      els = document.getElementsByClassName('image');
      sls = Array(els.length);
      for (j=0; j<els.length;++j) {
          sls[j]=els[j].outerHTML  
      };

We can then add this html to a new element in the document, which we will locate in MMA:

      el=document.createElement('div');
      el.id='myUniqueID';
      el.textContent=JSON.stringify(sls);
      document.children[0].appendChild(el);

We start a new session and load the required page

In[42]:= session = StartWebSession[];
In[43]:= WebExecute[session,"OpenPage" -> "https://imgur.com/gallery/uGNoByn"];

Now, we execute the code from above on the page:

In[44]:= js = "(function() {
      els = document.getElementsByClassName('image');
      sls = Array(els.length);
      for (j=0; j<els.length;++j) {
          sls[j]=els[j].outerHTML  
      };
      el=document.createElement('div');
      el.id='myUniqueID';
      el.textContent=JSON.stringify(sls);
      document.children[0].appendChild(el);
      })()";
In[45]:= WebExecute[session, "JavascriptExecute" -> js];

Now using MMA we extract the newly created element and get the html out of it:

In[46]:= els = 
  WebExecute[session, 
   "LocateElements" -> "CSSSelector" -> "#myUniqueID"];
In[47]:= WebExecute[session, "ElementText" -> els]
Out[47]= {"[\"<div class=\\\"image post-image\\\"><img alt=\\\"\\\" \
src=\\\"//i.imgur.com/qE3kW4Z.jpg\\\" style=\\\"max-width: 100%; \
min-height: 574px;\\\" original-title=\\\"\\\"></div>\"]"}

Now that the html is in MMA it is a matter of munging the required part out of it. Due to the vague nature of the question, this will be left as an exercise to the reader.

$\endgroup$
  • $\begingroup$ Thanks for the input. It will take me a bit to work through the javascript. The LocateElements tool is so useful in WebSession I hate to have to do the whole search process in another language just to bring it back into Wolfram. But perhaps that is the only way. $\endgroup$ – kickert Jan 25 at 14:53
  • 1
    $\begingroup$ I know what you mean about the different language - though the code is not complex if you can stomach it. PS, If you want a more generic selector that's analogous to the CSSSelector, take a look at document.querySelector at developer.mozilla.org/en-US/docs/Web/API/Document/querySelector` - then you won't have to do the whole array assignment business. Alternatively there's document.querySelectorAll which also returns a collection. $\endgroup$ – Shwouchk Jan 26 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.