Web Browsing Automation: how to expand all comments under a blog post?

Question

I wish to expand all the comments under a LiveJournal post for the purposes of saving the complete thread of comments. According to LiveJournal's FAQ when a post gets more than 50 comments they are automatically collapsed and LiveJournal doesn't provide an easy way to expand them: there is no Expand All button and the user must stupidly click on the every Expand button on the web page in order to get finally all the comments expanded. I wish to automate this process using either WebUnit or directly through NETLink using COM/.NET interface of Chrome (or other browser).

As an example post with collapsed comments I suggest this (but feel free to take any other post if you wish):

http://suggestions.livejournal.com/825341.html

---

So far I have tried WebUnit as follows without luck:

Needs["WebUnit`"]
InstallWebUnit[]
StartWebSession[];

OpenWebPage["http://suggestions.livejournal.com/825341.html"]
ClickElement[Id["t12335869"]]

The Id "t12335869" in the above I have found via inspection of the HTML source of the first Expand button on that web page:

 <li class=" b-leaf-actions-item b-leaf-actions-expandchilds   " data-tid="t12335869">   <a target="_self" href="http://suggestions.livejournal.com/825341.html?thread=12335869#t12335869" rel="nofollow" class="b-pseudo">Expand</a>   </li>

I have tried also interfacing Internet Explorer via the COM interface but haven't found a way to make the click[] method working:

Needs["NETLink`"];
InstallNET[];
ie = CreateCOMObject["InternetExplorer.Application"];
ie@visible = True;
ie@Navigate["http://suggestions.livejournal.com/825341.html"];
doc = ie@Document;
While[ie@Busy || doc@readyState =!= "complete", Pause[.1]];
all = doc@all;
elems = all@tags@"a";
Table[If[elems@item[i]@innerText === "Expand", elems@item[i]@click[]; 
  i, ## &[]], {i, 0, elems@length - 1}] // Length

50

From the above it is seen that 50 hyperlinks with text "Expand" are found. But executing click[] on them changes nothing.

---

How can I automate clicking the Expand buttons with Mathematica?

Answer 1

Here is one way to do it, using WebUnit:

Needs["WebUnit`"];
InstallWebUnit[];
StartWebSession[];
OpenWebPage["http://suggestions.livejournal.com/825341.html"];

Then, use JavascriptExecute to get all the links in the document and select only the ones that have the text EXPAND in them:

expandLinks = Flatten @ JavascriptExecute["
   var links = document.links;
   var result = []; 
   cnt=0; 
   for(i=0; i<links.length; i++) { 
    if( links[i].text=='Expand' ) { 
     result[cnt++] = links[i]; 
     } 
    };
   return result"]

Finally, click on all the expandLinks:

ClickElement["ELEMENT" /. #] & /@ Take[expandLinks, All]

Answer 2

I have found a working workaround for the Internet Explorer COM automation (but I still don't understand why the straightforward click[] method shown in the question does not work).

The solution is to execute on the web page a Javascript code using the javascript: protocol. In this post Jefferson Scher suggested a very efficient clickEm Javascript function that clicks only on the top level EXPAND buttons which expand multiple children collapsed comments. His solution also has an advantage of being independent from the language settings of LiveJournal (the EXPAND buttons have different localized names depending on the language settings). Here is a bookmarklet with his code:

javascript:(function(){var elinks = document.querySelectorAll('.b-tree-twig-1 .b-leaf-actions-expandchilds > a'); for (var i=0; i<elinks.length; i++){elinks[i].click();}})();

Here is complete COM automation code:

Needs["NETLink`"];
InstallNET[];
ie = CreateCOMObject["InternetExplorer.Application"];
ie@visible = True;
ie@Navigate["http://suggestions.livejournal.com/825341.html"];
(* waiting while post is being loaded *)
While[ie@Busy, Pause[.1]];
ie@Navigate[
  "javascript:(function(){var elinks =     \
document.querySelectorAll('.b-tree-twig-1 .b-leaf-actions-expandchilds > a'); \
    for (var i=0; i<elinks.length; i++){elinks[i].click();}})();"]
doc = ie@Document;
(* waiting while expanded comments are being loaded *)
While[ie@Busy || doc@readyState =!= "complete", Pause[.1]];

(tested with Internet Explorer 11 on Windows 7 x64).