2
$\begingroup$

I'm trying to make web crawler with Mathematica.

 webcrawler[rooturl_, depth_] := 
   Flatten[
     Rest[
       NestList[
         Union[Flatten[
           Thread[# -> Import[#, "Hyperlinks", "ServerAuthentication" -> False ]] & 
            /@ Last /@ #]] &, 
         {"" -> rooturl}, depth]]];

It's working well. The only problem is that whenever there is a link with user and password authentication, a box is poping up and I need to click Cancel. Is there any way to do it automatically?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ URLFetch has an option "DisplayProxyDialog" -> False but I don't know if that's what you are after since a working example is missing. $\endgroup$
    – Kuba
    Commented Jun 17, 2016 at 11:29
  • $\begingroup$ Do you mind posting more of the code that you have used? $\endgroup$
    – William
    Commented Aug 29, 2016 at 1:00

0

Reset to default

Browse other questions tagged or ask your own question.