4
$\begingroup$

I've successfully and fruitfully imported data from web sites before, but now seem to have a particularly troublesome site that I would like to access through Mathematica on my MacBook.

Import[site="https://auctionfiling.fcc.gov/form175/search175/index.htm"]

returns the error:

FetchURL::httperr: The request to URL 
https://auctionfiling.fcc.gov/form175/search175/index.htm was not successful.
The server returned the HTTP status code 500 ("Internal Server Error").

Exploring further, the command URLExecute[site] returns an long exception report that includes the text:

<cflog text="#Variables.errorTemplate#: UnsupportedBrowser The
browser you are using is not supported. <br>The error occurred on
line 32." type="error">

Further text indicates that the site requires either: Microsoft Internet Explorer 5.5 (or higher), Netscape 6 (or higher), Opera 6 (or higher), or Mozilla. Though I've successfully used it with Safari, Chrome and Firefox.

Is there a way around this? I am wanting to pull down many thousands of FCC license records through this approach.

Alternatively, perhaps there is an approach of running an external browser linked with Mathematica?

Thank you for any suggestions.

Steve

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ To spoof the user agent string: URLRead[HTTPRequest["https://auctionfiling.fcc.gov/form175/search175/index.htm", <|"UserAgent" -> "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>], "Body"] or whatever other user agent strings you want to use. $\endgroup$
    – J. M.'s lack of A.I. ♦
    May 17, 2017 at 15:29
  • $\begingroup$ @J.M. I think that should be an answer. $\endgroup$
    – C. E.
    May 17, 2017 at 16:15
  • $\begingroup$ @J.M. With your code Mathematica sends User-Agent: Wolfram HTTPClient 11.1 (Mathematica 11.1.1). The same is with version 11.0.1 (the string is User-Agent: Wolfram HTTPClient 11.). Looks like currently this functionality is broken. :( $\endgroup$
    – Alexey Popkov
    May 18, 2017 at 15:29
  • $\begingroup$ @Alexey, for reference, how did you test that the spoof is not working? I just tried in 11.1.1, and I was able to get through the browser-based block. $\endgroup$
    – J. M.'s lack of A.I. ♦
    May 18, 2017 at 15:41
  • 1
    $\begingroup$ @J.M. I've found that the failure was caused by hidden Unicode characters inside of the URL copied from your first comment (with the second comment the situation is even worse what helped me to figure out what happens). After fixing the URL now it works for me. But user-agent according to HandyCache is still User-Agent: Wolfram HTTPClient 11.1... May be "UserAgent" is used after the secure connection with the server is established? HandyCache reports only the header of initial CONNECT request. $\endgroup$
    – Alexey Popkov
    May 18, 2017 at 16:29

2 Answers 2

Reset to default
2
$\begingroup$

Posting J.M.'s comment as an answer, you can specify the user agent string in HTTPRequest directly via "UserAgent" (here it is for Firefox 40.1 taken from this site):

URLRead[HTTPRequest[
  "https://auctionfiling.fcc.gov/form175/search175/index.htm", <|
   "UserAgent" -> 
    "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>]]

output

(note that the code in the comments is broken due to hidden Unicode characters).

It is worth to know that

The default "UserAgent" is typically "Wolfram HTTP Client xxxx", where xxxx is a version number.

Improve this answer
$\endgroup$
4
  • $\begingroup$ How odd; it seems posting the code as a comment does munge the URL. Tsk... $\endgroup$
    – J. M.'s lack of A.I. ♦
    May 18, 2017 at 16:33
  • $\begingroup$ Alexey Popkov, Thank you for the very helpful insights. I was not aware of the HTTPRequest function. This has been very helpful for more days of progress. I'm now confused on a deeper level. I'm not at all knowledgable about XML but I've used hints from the answer to another question similar to this ( mathematica.stackexchange.com/questions/2362/…) But I'm still unable to set parameters in my request with Import[$req, "Data", "RequestMethod" -> "Post", "RequestParameters" -> {aucSelect"->1002}]. I'm now floundering, but grateful. $\endgroup$
    – Stephen Wilkus
    May 19, 2017 at 22:57
  • $\begingroup$ @Stephen That's entirely different question, you shouldn't discuss it here. Instead, create another well-formed question on this issue. $\endgroup$
    – Alexey Popkov
    May 20, 2017 at 1:54
  • $\begingroup$ Quite right. Thank you for your guidance. $\endgroup$
    – Stephen Wilkus
    May 20, 2017 at 2:16
0
$\begingroup$

Alternatively, perhaps there is an approach of running an external browser linked with Mathematica?

This seems to be possible since v12.0, using WebExecute. You can find it in the new-features page of v12, under "Controlling Web Browsers" and described shortly as "execute commands in a web browser".

I don't know whether this would have been sufficient for your purposes (and, I would suspect, it's too late for your original task). But it's an interesting option to consider for similar problems.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.