webscraping: Wolfram notebook vs browser

Question

In the Netherlands there is a website Funda.nl There you can find all houses for sale or for rent. I used webscraping as a tool to collect data for analyse-purpose. The script I used is like:

fundahuur1 = {}; For [pag = 1, pag <= 10, pag++,

     fundahuur1 = 
      Import["https://www.funda.nl/huur/amsterdam/+1km/p" <> 
        ToString[pag], "Hyperlinks"];
     AppendTo[fundahuur1, 
      Tally[Select[
         StringCases[fundahuur1, 
          "https://www.funda.nl/huur/amsterdam/" ~~ __ ~~ "-" ~~ 
           DigitCharacter ..], # != {} &]][[All, 1]]];
     ClearCookies[All];
     ]

recently the script doesn't work anymore. Maybe because the server recognise my request. When I use the same URL

https://www.funda.nl/huur/amsterdam/+1km/p1

in my Chrome browser I get the hyperlinks. So the funda-server makes a difference between my Chrome browser and my Wolfram-notebook.

My question are: 1. is there a solution for this problem so I can get my data 2. can I simulate a chrome-browser in my notebook

Answer 1

You can actually start and control a web browser session from Mathematica using an experimental implementation for WebDriver, Since Mathematica 11.3. Look at this answer

An example

Module[
 {
  session = StartExternalSession["WebDriver-Chrome"],
  iws, chromedo, img, links
  },
 chromedo[cmd_] := ExternalEvaluate[session, cmd];
 Pause[1];
 iws = ExternalEvaluateWebDriver`Private`websession[];
 Pause[1];(*Time to load chrome*)
 chromedo["OpenWebPage" -> 
   "https://www.funda.nl/huur/amsterdam/+1km/p"];
 Pause[3];(*Time to load the page*)
 links = Union@WebUnit`PageLinks[iws];
 DeleteObject[session];
 links
 ]

Be nice and try not to abuse the site, it's likely that you will be breaking the rules by gathering this information.