2
$\begingroup$

I can get the hyperlinks of the Google search result of "Wolfram" by using the following code:

Import["https://www.google.nl/search?q=Wolfram", "Hyperlinks"]

Now I want to use the same trick. I want to have the arXiv ID (or just the hyperlinks) of the papers referenced by the arXiv paper "1609.05206". So I used the below code:

Import["https://ui.adsabs.harvard.edu/abs/arXiv:1609.05206/references", "Hyperlinks"]

But what I got this time is just {"//ui.adsabs.harvard.edu"}, which is not what I want.

How can I get the clickable hyperlinks of the page (or just get those papers' arXiv ID)?

Thanks

Improve this question
$\endgroup$
3
  • $\begingroup$ Another option for this kind of dynamic site is to use the browser tools to see what queries the page is making and use that as your source - in this case, something like ui.adsabs.harvard.edu/v1/search/… $\endgroup$
    – Carl Lange
    Commented Aug 19, 2019 at 6:19
  • $\begingroup$ Thanks. If I clicked the link directly, it shows the referenced papers. But if I run Import["https://ui.adsabs.harvard.edu/v1/search/query?fl=title%2Cbibcode%2Cauthor%2Ckeyword%2Cpub%2Caff%2Cvolume%2Cyear%2C%5Bcitations%5D%2Cproperty%2Cpubdate%2Cabstract%2Cesources%2Cdata&q=references(identifier%3A2017Prama..89...80S)&rows=25&sort=first_author%20asc&start=0", "Text"], it requires me to input username and password... (even if I registered for the ADS' account, the username and password also doesn't work). Any ideas how to by-pass the username/password request? $\endgroup$
    – H42
    Commented Aug 19, 2019 at 17:28
  • $\begingroup$ In this case my expectation is that if you set the Host header to the one the site passes in its request, it may bypass that requirement. However, this is almost certainly breaking the terms of service of the site. $\endgroup$
    – Carl Lange
    Commented Aug 19, 2019 at 17:46

1 Answer 1

Reset to default
3
$\begingroup$

I just found an answer to my own question. I'm not sure it's a good answer since it would open a new browser. I would prefer to do it headlessly.

session = StartWebSession[]; 
myLink = "https://ui.adsabs.harvard.edu/abs/arXiv:1609.05206/references";
WebExecute[session, "OpenPage" -> myLink];
Pause[3]; (*pause for 3 seconds to let JavaScript run*)
links001 = WebExecute["PageHyperlinks"]
links002=DeleteDuplicates@links001

And then search inside links002 for specific keywords to get the reference.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You can pass Visible -> False to StartWebSession to do this headlessly. $\endgroup$
    – Carl Lange
    Commented Aug 19, 2019 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.