5
$\begingroup$

In the Netherlands there is a website Funda.nl There you can find all houses for sale or for rent. I used webscraping as a tool to collect data for analyse-purpose. The script I used is like:

fundahuur1 = {}; For [pag = 1, pag <= 10, pag++,

     fundahuur1 = 
      Import["https://www.funda.nl/huur/amsterdam/+1km/p" <> 
        ToString[pag], "Hyperlinks"];
     AppendTo[fundahuur1, 
      Tally[Select[
         StringCases[fundahuur1, 
          "https://www.funda.nl/huur/amsterdam/" ~~ __ ~~ "-" ~~ 
           DigitCharacter ..], # != {} &]][[All, 1]]];
     ClearCookies[All];
     ]

recently the script doesn't work anymore. Maybe because the server recognise my request. When I use the same URL

https://www.funda.nl/huur/amsterdam/+1km/p1

in my Chrome browser I get the hyperlinks. So the funda-server makes a difference between my Chrome browser and my Wolfram-notebook.

My question are: 1. is there a solution for this problem so I can get my data 2. can I simulate a chrome-browser in my notebook

$\endgroup$
  • 2
    $\begingroup$ You can simulate a Chrome browser by using the new Chrome driver interface (example), but the usual solution to your problem is to do what's known as browser spoofing. This entails opening up the developer tool in the Chrome browser and looking up what the HTTP requests look like. Then you set the same parameters, for things such as user agent, for your HTTP request in Mathematica, so that the HTTP requests are the same. The idea is that if the HTTP request is identical then there is no difference for the receiving server. $\endgroup$ – C. E. Oct 22 '18 at 8:22
7
$\begingroup$

You can actually start and control a web browser session from Mathematica using an experimental implementation for WebDriver, Since Mathematica 11.3. Look at this answer

An example

Module[
 {
  session = StartExternalSession["WebDriver-Chrome"],
  iws, chromedo, img, links
  },
 chromedo[cmd_] := ExternalEvaluate[session, cmd];
 Pause[1];
 iws = ExternalEvaluateWebDriver`Private`websession[];
 Pause[1];(*Time to load chrome*)
 chromedo["OpenWebPage" -> 
   "https://www.funda.nl/huur/amsterdam/+1km/p"];
 Pause[3];(*Time to load the page*)
 links = Union@WebUnit`PageLinks[iws];
 DeleteObject[session];
 links
 ]

Be nice and try not to abuse the site, it's likely that you will be breaking the rules by gathering this information.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.