Getting data using webscraping (Obtaining links from search engine with redirect)

Question

In the past I have used the function Import in relation with "Hyperlink" to collect hyperlinks and data from the internet.

For example:

Import["https://www.google.nl/search?q=Wolfram", "Hyperlinks"]

gives a list with all hyperlinks. In the next step I can use these hyperlinks to scrape the web pages. This works fine.

When I want to use the search engine DuckDuckGo, like:

Import["https://duckduckgo.com/?q=%22Wolfram%22&t=h_&ia=web", "Hyperlinks"]

the output is:

{ "https://duckduckgo.com//?t=h_", "https://duckduckgo.com/html/?q=%22Wolfram%22" }

When I go to the webpage of DuckDuckGo and type a question, then I get a full list of search results. My question is:

How can I collect this data using Wolfram Language?

Answer 1

Let's step back and do something simpler:

Import["https://duckduckgo.com/?q=%22Wolfram%22&t=h_&ia=web"]
(* "Ignore this box please.

DuckDuckGo

You are being redirected to the non-JavaScript site.
Click here if it doesn't happen automatically." *)

Oh, you are been redirected! To where?

Import[
 "https://duckduckgo.com/?q=%22Wolfram%22&t=h_&ia=web",
 "Hyperlinks"
 ]
(* {"https://duckduckgo.com//?t=h_", "https://duckduckgo.com/html/?q=%22Wolfram%22"} *)

Aha!, here "https://duckduckgo.com/html/?q=%22Wolfram%22"

Then we better use "https://duckduckgo.com/html/?q=" directly

getLinks[query_String] := Union@Import[
   StringTemplate["https://duckduckgo.com/html/?q=%22``%22\""]@
    URLEncode[query]
   , "Hyperlinks"
   ]


getLinks["Mathematica"]