# Importing from a web page that seems to expect specific browsers

I've successfully and fruitfully imported data from web sites before, but now seem to have a particularly troublesome site that I would like to access through Mathematica on my MacBook.

Import[site="https://auctionfiling.fcc.gov/form175/search175/index.htm"]


returns the error:

FetchURL::httperr: The request to URL
https://auctionfiling.fcc.gov/form175/search175/index.htm was not successful.
The server returned the HTTP status code 500 ("Internal Server Error").


Exploring further, the command URLExecute[site] returns an long exception report that includes the text:

<cflog text="#Variables.errorTemplate#: UnsupportedBrowser The
browser you are using is not supported. <br>The error occurred on
line 32." type="error">


Further text indicates that the site requires either: Microsoft Internet Explorer 5.5 (or higher), Netscape 6 (or higher), Opera 6 (or higher), or Mozilla. Though I've successfully used it with Safari, Chrome and Firefox.

Is there a way around this? I am wanting to pull down many thousands of FCC license records through this approach.

Alternatively, perhaps there is an approach of running an external browser linked with Mathematica?

Thank you for any suggestions.

Steve

• To spoof the user agent string: URLRead[HTTPRequest["https://auctionfiling.fcc.gov/form175/search175/index.htm", <|"UserAgent" -> "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>], "Body"] or whatever other user agent strings you want to use. May 17 '17 at 15:29
• @J.M. I think that should be an answer. May 17 '17 at 16:15
• @J.M. With your code Mathematica sends User-Agent: Wolfram HTTPClient 11.1 (Mathematica 11.1.1). The same is with version 11.0.1 (the string is User-Agent: Wolfram HTTPClient 11.). Looks like currently this functionality is broken. :( May 18 '17 at 15:29
• @Alexey, for reference, how did you test that the spoof is not working? I just tried in 11.1.1, and I was able to get through the browser-based block. May 18 '17 at 15:41
• @J.M. I've found that the failure was caused by hidden Unicode characters inside of the URL copied from your first comment (with the second comment the situation is even worse what helped me to figure out what happens). After fixing the URL now it works for me. But user-agent according to HandyCache is still User-Agent: Wolfram HTTPClient 11.1... May be "UserAgent" is used after the secure connection with the server is established? HandyCache reports only the header of initial CONNECT request. May 18 '17 at 16:29

Posting J.M.'s comment as an answer, you can specify the user agent string in HTTPRequest directly via "UserAgent" (here it is for Firefox 40.1 taken from this site):

URLRead[HTTPRequest[
"https://auctionfiling.fcc.gov/form175/search175/index.htm", <|
"UserAgent" ->
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>]]


(note that the code in the comments is broken due to hidden Unicode characters).

It is worth to know that

The default "UserAgent" is typically "Wolfram HTTP Client xxxx", where xxxx is a version number.

• How odd; it seems posting the code as a comment does munge the URL. Tsk... May 18 '17 at 16:33
• Alexey Popkov, Thank you for the very helpful insights. I was not aware of the HTTPRequest function. This has been very helpful for more days of progress. I'm now confused on a deeper level. I'm not at all knowledgable about XML but I've used hints from the answer to another question similar to this ( mathematica.stackexchange.com/questions/2362/…) But I'm still unable to set parameters in my request with Import[\$req, "Data", "RequestMethod" -> "Post", "RequestParameters" -> {aucSelect"->1002}]. I'm now floundering, but grateful. May 19 '17 at 22:57
• @Stephen That's entirely different question, you shouldn't discuss it here. Instead, create another well-formed question on this issue. May 20 '17 at 1:54
• Quite right. Thank you for your guidance. May 20 '17 at 2:16

Alternatively, perhaps there is an approach of running an external browser linked with Mathematica?

This seems to be possible since v12.0, using WebExecute. You can find it in the new-features page of v12, under "Controlling Web Browsers" and described shortly as "execute commands in a web browser".

I don't know whether this would have been sufficient for your purposes (and, I would suspect, it's too late for your original task). But it's an interesting option to consider for similar problems.