# Unable to use the Import[] function to get the clickable hyperlinks of a page

I can get the hyperlinks of the Google search result of "Wolfram" by using the following code:

Import["https://www.google.nl/search?q=Wolfram", "Hyperlinks"]


Now I want to use the same trick. I want to have the arXiv ID (or just the hyperlinks) of the papers referenced by the arXiv paper "1609.05206". So I used the below code:

Import["https://ui.adsabs.harvard.edu/abs/arXiv:1609.05206/references", "Hyperlinks"]


But what I got this time is just {"//ui.adsabs.harvard.edu"}, which is not what I want.

How can I get the clickable hyperlinks of the page (or just get those papers' arXiv ID)?

Thanks

• Another option for this kind of dynamic site is to use the browser tools to see what queries the page is making and use that as your source - in this case, something like ui.adsabs.harvard.edu/v1/search/… Aug 19, 2019 at 6:19
• Thanks. If I clicked the link directly, it shows the referenced papers. But if I run Import["https://ui.adsabs.harvard.edu/v1/search/query?fl=title%2Cbibcode%2Cauthor%2Ckeyword%2Cpub%2Caff%2Cvolume%2Cyear%2C%5Bcitations%5D%2Cproperty%2Cpubdate%2Cabstract%2Cesources%2Cdata&q=references(identifier%3A2017Prama..89...80S)&rows=25&sort=first_author%20asc&start=0", "Text"], it requires me to input username and password... (even if I registered for the ADS' account, the username and password also doesn't work). Any ideas how to by-pass the username/password request?
– H42
Aug 19, 2019 at 17:28
• In this case my expectation is that if you set the Host header to the one the site passes in its request, it may bypass that requirement. However, this is almost certainly breaking the terms of service of the site. Aug 19, 2019 at 17:46

I just found an answer to my own question. I'm not sure it's a good answer since it would open a new browser. I would prefer to do it headlessly.

session = StartWebSession[];

And then search inside links002 for specific keywords to get the reference.
• You can pass Visible -> False to StartWebSession to do this headlessly. Aug 19, 2019 at 6:21