3
$\begingroup$

I've successfully and fruitfully imported data from web sites before, but now seem to have a particularly troublesome site that I would like to access through Mathematica on my MacBook.

Import[site="https://auctionfiling.fcc.gov/form175/search175/index.htm"]

returns the error:

FetchURL::httperr: The request to URL 
https://auctionfiling.fcc.gov/form175/search175/index.htm was not successful.
The server returned the HTTP status code 500 ("Internal Server Error").

Exploring further, the command URLExecute[site] returns an long exception report that includes the text:

<cflog text="#Variables.errorTemplate#: UnsupportedBrowser The
browser you are using is not supported. <br>The error occurred on
line 32." type="error">

Further text indicates that the site requires either: Microsoft Internet Explorer 5.5 (or higher), Netscape 6 (or higher), Opera 6 (or higher), or Mozilla. Though I've successfully used it with Safari, Chrome and Firefox.

Is there a way around this? I am wanting to pull down many thousands of FCC license records through this approach.

Alternatively, perhaps there is an approach of running an external browser linked with Mathematica?

Thank you for any suggestions.

Steve

$\endgroup$
  • 2
    $\begingroup$ To spoof the user agent string: URLRead[HTTPRequest["https://auctionfiling.fcc.gov/form175/search175/index.htm", <|"UserAgent" -> "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>], "Body"] or whatever other user agent strings you want to use. $\endgroup$ – J. M. will be back soon May 17 '17 at 15:29
  • $\begingroup$ @J.M. I think that should be an answer. $\endgroup$ – C. E. May 17 '17 at 16:15
  • $\begingroup$ @J.M. With your code Mathematica sends User-Agent: Wolfram HTTPClient 11.1 (Mathematica 11.1.1). The same is with version 11.0.1 (the string is User-Agent: Wolfram HTTPClient 11.). Looks like currently this functionality is broken. :( $\endgroup$ – Alexey Popkov May 18 '17 at 15:29
  • $\begingroup$ @Alexey, for reference, how did you test that the spoof is not working? I just tried in 11.1.1, and I was able to get through the browser-based block. $\endgroup$ – J. M. will be back soon May 18 '17 at 15:41
  • 1
    $\begingroup$ @J.M. I've found that the failure was caused by hidden Unicode characters inside of the URL copied from your first comment (with the second comment the situation is even worse what helped me to figure out what happens). After fixing the URL now it works for me. But user-agent according to HandyCache is still User-Agent: Wolfram HTTPClient 11.1... May be "UserAgent" is used after the secure connection with the server is established? HandyCache reports only the header of initial CONNECT request. $\endgroup$ – Alexey Popkov May 18 '17 at 16:29
1
$\begingroup$

Posting J.M.'s comment as an answer, you can specify the user agent string in HTTPRequest directly via "UserAgent" (here it is for Firefox 40.1 taken from this site):

URLRead[HTTPRequest[
  "https://auctionfiling.fcc.gov/form175/search175/index.htm", <|
   "UserAgent" -> 
    "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"|>]]

output

(note that the code in the comments is broken due to hidden Unicode characters).

It is worth to know that

The default "UserAgent" is typically "Wolfram HTTP Client xxxx", where xxxx is a version number.

$\endgroup$
  • $\begingroup$ How odd; it seems posting the code as a comment does munge the URL. Tsk... $\endgroup$ – J. M. will be back soon May 18 '17 at 16:33
  • $\begingroup$ Alexey Popkov, Thank you for the very helpful insights. I was not aware of the HTTPRequest function. This has been very helpful for more days of progress. I'm now confused on a deeper level. I'm not at all knowledgable about XML but I've used hints from the answer to another question similar to this ( mathematica.stackexchange.com/questions/2362/…) But I'm still unable to set parameters in my request with Import[$req, "Data", "RequestMethod" -> "Post", "RequestParameters" -> {aucSelect"->1002}]. I'm now floundering, but grateful. $\endgroup$ – Stephen Wilkus May 19 '17 at 22:57
  • $\begingroup$ @Stephen That's entirely different question, you shouldn't discuss it here. Instead, create another well-formed question on this issue. $\endgroup$ – Alexey Popkov May 20 '17 at 1:54
  • $\begingroup$ Quite right. Thank you for your guidance. $\endgroup$ – Stephen Wilkus May 20 '17 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.