I've successfully and fruitfully imported data from web sites before, but now seem to have a particularly troublesome site that I would like to access through Mathematica on my MacBook.
Import[site="https://auctionfiling.fcc.gov/form175/search175/index.htm"]
returns the error:
FetchURL::httperr: The request to URL
https://auctionfiling.fcc.gov/form175/search175/index.htm was not successful.
The server returned the HTTP status code 500 ("Internal Server Error").
Exploring further, the command URLExecute[site]
returns an long exception report that includes the text:
<cflog text="#Variables.errorTemplate#: UnsupportedBrowser The
browser you are using is not supported. <br>The error occurred on
line 32." type="error">
Further text indicates that the site requires either: Microsoft Internet Explorer 5.5 (or higher), Netscape 6 (or higher), Opera 6 (or higher), or Mozilla. Though I've successfully used it with Safari, Chrome and Firefox.
Is there a way around this? I am wanting to pull down many thousands of FCC license records through this approach.
Alternatively, perhaps there is an approach of running an external browser linked with Mathematica?
Thank you for any suggestions.
Steve
URLRead[HTTPRequest["https://auctionfiling.fcc.gov/form175/search175/index.htm", <|"UserAgent" -> "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>], "Body"]
or whatever other user agent strings you want to use. $\endgroup$User-Agent: Wolfram HTTPClient 11.1
(Mathematica 11.1.1). The same is with version 11.0.1 (the string isUser-Agent: Wolfram HTTPClient 11.
). Looks like currently this functionality is broken. :( $\endgroup$User-Agent: Wolfram HTTPClient 11.1
... May be"UserAgent"
is used after the secure connection with the server is established? HandyCache reports only the header of initialCONNECT
request. $\endgroup$