10
$\begingroup$

I have been using WebSession and WebExecute to pull in information from a website. Using LocateElements withCSSSelector has allowed me to easily identify sections on a website and pull in the text. However, I am struggling to figure out a way to extract the underlying hyperlink from a linked image. I can identify the WebElementObject but I am at a loss for how to get the link from it. Perhaps it isn't possible with the tools I am using, or perhaps I am overlooking something simple. Perhaps I could use a JavaScriptExecute argument?

My solution right now is to pull in all the page hyperlinks and use several other functions to figure out which is the appropriate link.

Here is the code to pull in the text from the elements. In this case, I am looking at the links in the footer of the wolfram website. What I want is the hyperlinks behind that text.

session = StartWebSession["Chrome"];
WebExecute[session, "OpenPage" -> "www.wolfram.com"];
WebExecute[session, 
 "ElementText" -> 
  WebExecute[session, 
   "LocateElements" -> "CSSSelector" -> "#_footer-bc > ul > li > a"]]

{"Legal", "Privacy Policy", "Site Map", "WolframAlpha.com", "WolframCloud.com"}

It would be great to pull any of the attributes from a WebElement that you have already identified as that could come in handy in a lot of ways.

$\endgroup$
2
  • $\begingroup$ It would help if you would provide some example code and and a sample html in which you would specify what part of what element you're looking for. Otherwise this is too abstract... $\endgroup$
    – Shwouchk
    Commented Jan 25, 2020 at 2:42
  • $\begingroup$ Just added some example code on identifying elements and then pulling text. $\endgroup$
    – kickert
    Commented Jan 25, 2020 at 14:37

3 Answers 3

7
$\begingroup$

Here is a way with no JavaScript. Using the getAttribute function from this answer (by @swish)

getAttribute[element_WebElementObject, attribute_String] := 
 getAttribute[$CurrentWebSession, element, attribute]

getAttribute[session_WebSessionObject, element_WebElementObject, attribute_String] := 
 With[{sessionInfo = session /@ {"SessionID", "Browser", "URL"}}, 
  WebUnit`Private`attribute[sessionInfo, element["ElementId"], attribute]]

you can do:

StartWebSession["Chrome"];
WebExecute["OpenPage" -> "www.wolfram.com"];
imgs = WebExecute["LocateElements" -> "CSSSelector" -> "#_footer-bc > ul > li > a"];
getAttribute[#, "href"]&/@imgs

to get:

{http://www.wolfram.com/legal/?source=footer,http://www.wolfram.com/legal/privacy/wolfram/?source=footer,http://www.wolfram.com/site-map/?source=footer,http://www.wolframalpha.com/?source=footer,https://www.wolframcloud.com/?source=footer}

$\endgroup$
2
  • 1
    $\begingroup$ This is excellent. Even though I had already awarded the bounty due to time constraints, this is the answer that needs attention. It is exactly what I was looking for. Was finally able to deploy it today and it worked perfectly. Would you consider adding the original code from the previous post to this one (with proper credit) so people can get the whole answer in one place? $\endgroup$
    – kickert
    Commented May 1, 2020 at 15:08
  • 1
    $\begingroup$ @kickert nice that it was useful. Suggestion accepted. 👍🏻 $\endgroup$
    – Murta
    Commented May 1, 2020 at 15:53
5
+50
$\begingroup$

As stated in the comment, I think the question was not very well specified so it would be quite hard to give a good satisfactory answer. However, I did not know about WebExecute before and got curious about it. After playing around a bit and reading some docs, my conclusion is that there's no direct way to get the HTML or to manipulate the document using the WebElementObject objects returned by WebExecute.

However, a workaround presents itself in the form of the JavascriptExecute command. The idea is to use a javascript snippet to extract the sought after element(s) and then to add the entire html of said elements as the text of a new element with a given unique ID. Then we return to Mathematica where we find this new element and use ElementText to extract the html of the elements we extracted in JS. Since no HTML was provided I demonstrate on an arbitrary page and assume that OP would be able to narrow to the required element with javascript.

Here is sample code to extract an img tag from https://i.sstatic.net/Sm0rs.jpg:

First, here is a snippet of the original HTML found when inspecting the main image on that page:

<div class="image post-image"><img alt="" src="//i.imgur.com/qE3kW4Z.jpg" style="max-width: 100%; min-height: 574px;" original-title=""></div>

For the sake of this example we'll use the image class to find the appropriate elements and extract their html:

      els = document.getElementsByClassName('image');
      sls = Array(els.length);
      for (j=0; j<els.length;++j) {
          sls[j]=els[j].outerHTML  
      };

We can then add this html to a new element in the document, which we will locate in MMA:

      el=document.createElement('div');
      el.id='myUniqueID';
      el.textContent=JSON.stringify(sls);
      document.children[0].appendChild(el);

We start a new session and load the required page

In[42]:= session = StartWebSession[];
In[43]:= WebExecute[session,"OpenPage" -> "https://i.sstatic.net/Sm0rs.jpg"];

Now, we execute the code from above on the page:

In[44]:= js = "(function() {
      els = document.getElementsByClassName('image');
      sls = Array(els.length);
      for (j=0; j<els.length;++j) {
          sls[j]=els[j].outerHTML  
      };
      el=document.createElement('div');
      el.id='myUniqueID';
      el.textContent=JSON.stringify(sls);
      document.children[0].appendChild(el);
      })()";
In[45]:= WebExecute[session, "JavascriptExecute" -> js];

Now using MMA we extract the newly created element and get the html out of it:

In[46]:= els = 
  WebExecute[session, 
   "LocateElements" -> "CSSSelector" -> "#myUniqueID"];
In[47]:= WebExecute[session, "ElementText" -> els]
Out[47]= {"[\"<div class=\\\"image post-image\\\"><img alt=\\\"\\\" \
src=\\\"//i.imgur.com/qE3kW4Z.jpg\\\" style=\\\"max-width: 100%; \
min-height: 574px;\\\" original-title=\\\"\\\"></div>\"]"}

Now that the html is in MMA it is a matter of munging the required part out of it. Due to the vague nature of the question, this will be left as an exercise to the reader.

$\endgroup$
2
  • $\begingroup$ Thanks for the input. It will take me a bit to work through the javascript. The LocateElements tool is so useful in WebSession I hate to have to do the whole search process in another language just to bring it back into Wolfram. But perhaps that is the only way. $\endgroup$
    – kickert
    Commented Jan 25, 2020 at 14:53
  • 1
    $\begingroup$ I know what you mean about the different language - though the code is not complex if you can stomach it. PS, If you want a more generic selector that's analogous to the CSSSelector, take a look at document.querySelector at developer.mozilla.org/en-US/docs/Web/API/Document/querySelector` - then you won't have to do the whole array assignment business. Alternatively there's document.querySelectorAll which also returns a collection. $\endgroup$
    – Shwouchk
    Commented Jan 26, 2020 at 9:23
2
$\begingroup$
session = StartWebSession["Chrome"];

WebExecute[session, "OpenPage" -> "www.wolfram.com"];
import = WebExecute[
          "LocateElements" -> "CSSSelector" -> "#_footer-bc >ul>li>a"];

tags = WebExecute[session, "ElementTag" -> import];

WebExecute[
  "ElementAttribute" -> {import[[#]], "href"}] & /@ Range[Length@tags]

(* 
  {"https://www.wolfram.com/legal/?source=footer", 
   "https://www.wolfram.com/legal/privacy/wolfram/?source=footer", 
   "https://www.wolfram.com/site-map/?source=footer", 
   "https://www.wolframalpha.com/?source=footer", 
   "https://www.wolframcloud.com/?source=footer" } *) 

ElementAttribute

Victor K has drawn attention to the very useful command, ElementAttribute, which was undocumented at the time of his post.

However, it now documented

"WebExecute supports the following commands related to elements of webpages"

"ElementAttribute" ->{elem,"attr"}  // an attribute of an element
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.