6
$\begingroup$

Version 11 introduces URLRead (and others) to replace URLFetch, but I have run into a strange problem while trying to switch to the new function.

These two both work:

URLFetch["http://www.nottingham.ac.uk/physics/"]
URLRead["http://www.nottingham.ac.uk/physics/"]

But whereas

URLFetch["http://www.nottingham.ac.uk/~ppzphy/"]

successfully loads another page on the same server, this one

URLRead["http://www.nottingham.ac.uk/~ppzphy/"]

gives a 404 error (on my machine, running Mathematica 11.0 on Windows 7).

I had suspected that the problem was the ~ in the second URL, but replacing it by %7E does not work, and other pages with ~ can be loaded:

URLRead["https://www.cs.tut.fi/~jkorpela/tilde.html"]

Other functions such as Import work fine with these addresses.


Update to summarize the answers: This seems to be a combination of a problem with the server setup, so that ~ and %7E are treated as distinct, and inconsistent URLEncode-ing behavior of URLRead.

$\endgroup$
  • $\begingroup$ I can't reproduce the error. Both URLFetch and URLRead works. (Fetch returns the page, Read returns 200-OK, clicking the green button shows the exact same redirect page source) (Mathematica 11.0 from EU/Hungary) $\endgroup$ – Gyebro Jan 13 '17 at 15:01
  • $\begingroup$ 404 here. Win10 V11. $\endgroup$ – Kuba Jan 13 '17 at 15:32
  • $\begingroup$ 404 here as well. MacOS 10.12.2, MMA v.11.0.1 $\endgroup$ – Cassini Jan 13 '17 at 18:08
3
$\begingroup$

Don't know how to automatically avoid that but it seems that everything which is considered "Path" or "Query" part of the url is automatically URLEncoded.

This works on Windows:

URLRead[<|"Scheme" -> "http", "Domain" -> "www.nottingham.ac.uk/~ppzphy/"|>, "Content"]

<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\">
<html>
<head>
<title>Your Page Title</title>
<meta http-equiv=\"REFRESH\" \
content=\"0;url=http://www.nottingham.ac.uk/physics/\"></HEAD>
<BODY>
Redirecting to<br><br>
<a href=\"http://www.nottingham.ac.uk/physics/\">http://www.\
nottingham.ac.uk/physics/</a>
</BODY>
</HTML>
$\endgroup$
4
$\begingroup$

This is an incomplete answer, but it should help. I can reproduce on Windows 7. Using Fiddler it is easy to see that URLFetch issues this request:

GET /~ppzphy/ HTTP/1.1

while URLRead issues a this one:

GET /%7Eppzphy/ HTTP/1.1

Unfortunately, I didn't find a way to make URLRead work for that address.

$\endgroup$
  • $\begingroup$ Thanks, that's helpful. URLFetch["http://www.nottingham.ac.uk/%7Eppzphy/"] returns the 404 error page. $\endgroup$ – Stephen Powell Jan 13 '17 at 16:20
  • 1
    $\begingroup$ A little off-topic, but opening http://www.nottingham.ac.uk/%7Eppzphy/ in Chrome and IE works correctly, but Firefox gives the 404 page. I think this is a problem with the way the server is set up. $\endgroup$ – Stephen Powell Jan 13 '17 at 16:31
3
$\begingroup$

It seems like building the URL like so helps:

URLRead[
 HTTPRequest[
  URL["http://www.nottingham.ac.uk"], 
  <|"Path" -> "~ppzphy/"|>
 ]
]

This appears to avoid having URLEncode interfere.

$\endgroup$
  • 2
    $\begingroup$ BTW, I would still report this to support@wolfram.com. At the very least its something they should be aware of. $\endgroup$ – chuy Jan 13 '17 at 16:26
  • 2
    $\begingroup$ Actually one could well argument that the problem is on the server side which to my understanding should be able to answer both cases ("~" should be escaped according to RFC1738 but the newer RFC2396 classifies it as an "unreserved" character which may or may not be escaped). The real problem on the Mathematica side for me seems to be that it doesn't behave consistently on different plattforms... $\endgroup$ – Albert Retey Jan 13 '17 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.