3
$\begingroup$

In the past I have used the function Import in relation with "Hyperlink" to collect hyperlinks and data from the internet.

For example:

Import["https://www.google.nl/search?q=Wolfram", "Hyperlinks"]

gives a list with all hyperlinks. In the next step I can use these hyperlinks to scrape the web pages. This works fine.

When I want to use the search engine DuckDuckGo, like:

Import["https://duckduckgo.com/?q=%22Wolfram%22&t=h_&ia=web", "Hyperlinks"]

the output is:

{ "https://duckduckgo.com//?t=h_", "https://duckduckgo.com/html/?q=%22Wolfram%22" }

When I go to the webpage of DuckDuckGo and type a question, then I get a full list of search results. My question is:

How can I collect this data using Wolfram Language?

$\endgroup$
5
$\begingroup$

Let's step back and do something simpler:

Import["https://duckduckgo.com/?q=%22Wolfram%22&t=h_&ia=web"]
(* "Ignore this box please.

DuckDuckGo

You are being redirected to the non-JavaScript site.
Click here if it doesn't happen automatically." *)

Oh, you are been redirected! To where?

Import[
 "https://duckduckgo.com/?q=%22Wolfram%22&t=h_&ia=web",
 "Hyperlinks"
 ]
(* {"https://duckduckgo.com//?t=h_", "https://duckduckgo.com/html/?q=%22Wolfram%22"} *)

Aha!, here "https://duckduckgo.com/html/?q=%22Wolfram%22"

Then we better use "https://duckduckgo.com/html/?q=" directly

getLinks[query_String] := Union@Import[
   StringTemplate["https://duckduckgo.com/html/?q=%22``%22\""]@
    URLEncode[query]
   , "Hyperlinks"
   ]


getLinks["Mathematica"]
$\endgroup$
  • $\begingroup$ Hi, this is a nice answer. Below the duckduckgo page you can go to the next page. Do you have a suggestion how to do that with a Mathematica Script? $\endgroup$ – Michiel van Mens Jul 12 '18 at 9:00
  • $\begingroup$ @MichielvanMens No, I don't. The next button is a "POST" form, it gets complicated. In any case, you are moving the goalpost to a place that is is no longer a Mathematica question. Let's stay on-topic. $\endgroup$ – rhermans Jul 12 '18 at 17:21
  • 1
    $\begingroup$ @MichielvanMens in Mathematica 11.3 look at "WebDriver-Chrome" which allows "ClickElement". $\endgroup$ – rhermans Jul 17 '18 at 17:50
  • $\begingroup$ @MichielvanMens I have posted a question and answer that may help you $\endgroup$ – rhermans Jul 26 '18 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.