7
$\begingroup$

I wish to expand all the comments under a LiveJournal post for the purposes of saving the complete thread of comments. According to LiveJournal's FAQ when a post gets more than 50 comments they are automatically collapsed and LiveJournal doesn't provide an easy way to expand them: there is no Expand All button and the user must stupidly click on the every Expand button on the web page in order to get finally all the comments expanded. I wish to automate this process using either WebUnit or directly through NETLink using COM/.NET interface of Chrome (or other browser).

As an example post with collapsed comments I suggest this (but feel free to take any other post if you wish):

http://suggestions.livejournal.com/825341.html


So far I have tried WebUnit as follows without luck:

Needs["WebUnit`"]
InstallWebUnit[]
StartWebSession[];

OpenWebPage["http://suggestions.livejournal.com/825341.html"]
ClickElement[Id["t12335869"]]

The Id "t12335869" in the above I have found via inspection of the HTML source of the first Expand button on that web page:

 <li class=" b-leaf-actions-item b-leaf-actions-expandchilds   " data-tid="t12335869">   <a target="_self" href="http://suggestions.livejournal.com/825341.html?thread=12335869#t12335869" rel="nofollow" class="b-pseudo">Expand</a>   </li>

I have tried also interfacing Internet Explorer via the COM interface but haven't found a way to make the click[] method working:

Needs["NETLink`"];
InstallNET[];
ie = CreateCOMObject["InternetExplorer.Application"];
ie@visible = True;
ie@Navigate["http://suggestions.livejournal.com/825341.html"];
doc = ie@Document;
While[ie@Busy || doc@readyState =!= "complete", Pause[.1]];
all = doc@all;
elems = all@tags@"a";
Table[If[elems@item[i]@innerText === "Expand", elems@item[i]@click[]; 
  i, ## &[]], {i, 0, elems@length - 1}] // Length
50

From the above it is seen that 50 hyperlinks with text "Expand" are found. But executing click[] on them changes nothing.


How can I automate clicking the Expand buttons with Mathematica?

$\endgroup$
7
$\begingroup$

Here is one way to do it, using WebUnit:

Needs["WebUnit`"];
InstallWebUnit[];
StartWebSession[];
OpenWebPage["http://suggestions.livejournal.com/825341.html"];

Then, use JavascriptExecute to get all the links in the document and select only the ones that have the text EXPAND in them:

expandLinks = Flatten @ JavascriptExecute["
   var links = document.links;
   var result = []; 
   cnt=0; 
   for(i=0; i<links.length; i++) { 
    if( links[i].text=='Expand' ) { 
     result[cnt++] = links[i]; 
     } 
    };
   return result"]

Finally, click on all the expandLinks:

ClickElement["ELEMENT" /. #] & /@ Take[expandLinks, All]
$\endgroup$
  • $\begingroup$ Thank you very much! It works but for some reason for the given webpage it expands all comments only on the second evaluation of the line ClickElement["ELEMENT" /. #] & /@ Take[expandLinks, All]. For some other web pages it requires 3 - 6 times to go through the cycle JavascriptExecute -> ClickElement in order to get all the comments expanded. Is is possible to overcome this problem and ensure that all the comments will be expanded? $\endgroup$ – Alexey Popkov Jun 28 '16 at 18:55
  • $\begingroup$ @AlexeyPopkov Web sites like this can be very dynamic: clicking one link affecting other links etc. So you may have to repeat the process, to be able to click on previously inaccessible links. Another problem could be that ClickElement is executing too fast for your browser to keep up with. In that case try adding small a small Pause[] between clicks. $\endgroup$ – Arnoud Buzing Jun 28 '16 at 20:06
  • $\begingroup$ Another question: is it possible to execute the Save As... dialog in Chrome? $\endgroup$ – Alexey Popkov Jun 28 '16 at 20:14
  • $\begingroup$ It might be possible, but I don't know how to do that. $\endgroup$ – Arnoud Buzing Jun 28 '16 at 21:43
  • $\begingroup$ The problem with not all comments being expanded can be solved nicely if we simply replace the line result[cnt++] = links[i]; with line links[i].click(); in the JavaScript code. ClickElement with this approach isn't needed. $\endgroup$ – Alexey Popkov Jun 29 '16 at 6:06
0
$\begingroup$

I have found a working workaround for the Internet Explorer COM automation (but I still don't understand why the straightforward click[] method shown in the question does not work).

The solution is to execute on the web page a Javascript code using the javascript: protocol. In this post Jefferson Scher suggested a very efficient clickEm Javascript function that clicks only on the top level EXPAND buttons which expand multiple children collapsed comments. His solution also has an advantage of being independent from the language settings of LiveJournal (the EXPAND buttons have different localized names depending on the language settings). Here is a bookmarklet with his code:

javascript:(function(){var elinks = document.querySelectorAll('.b-tree-twig-1 .b-leaf-actions-expandchilds > a'); for (var i=0; i<elinks.length; i++){elinks[i].click();}})();

Here is complete COM automation code:

Needs["NETLink`"];
InstallNET[];
ie = CreateCOMObject["InternetExplorer.Application"];
ie@visible = True;
ie@Navigate["http://suggestions.livejournal.com/825341.html"];
(* waiting while post is being loaded *)
While[ie@Busy, Pause[.1]];
ie@Navigate[
  "javascript:(function(){var elinks =     \
document.querySelectorAll('.b-tree-twig-1 .b-leaf-actions-expandchilds > a'); \
    for (var i=0; i<elinks.length; i++){elinks[i].click();}})();"]
doc = ie@Document;
(* waiting while expanded comments are being loaded *)
While[ie@Busy || doc@readyState =!= "complete", Pause[.1]];

(tested with Internet Explorer 11 on Windows 7 x64).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.