6
$\begingroup$

Version 11 introduces URLRead (and others) to replace URLFetch, but I have run into a strange problem while trying to switch to the new function.

These two both work:

URLFetch["http://www.nottingham.ac.uk/physics/"]
URLRead["http://www.nottingham.ac.uk/physics/"]

But whereas

URLFetch["http://www.nottingham.ac.uk/~ppzphy/"]

successfully loads another page on the same server, this one

URLRead["http://www.nottingham.ac.uk/~ppzphy/"]

gives a 404 error (on my machine, running Mathematica 11.0 on Windows 7).

I had suspected that the problem was the ~ in the second URL, but replacing it by %7E does not work, and other pages with ~ can be loaded:

URLRead["https://www.cs.tut.fi/~jkorpela/tilde.html"]

Other functions such as Import work fine with these addresses.

Update to summarize the answers: This seems to be a combination of a problem with the server setup, so that ~ and %7E are treated as distinct, and inconsistent URLEncode-ing behavior of URLRead.

Improve this question
$\endgroup$
3
  • $\begingroup$ I can't reproduce the error. Both URLFetch and URLRead works. (Fetch returns the page, Read returns 200-OK, clicking the green button shows the exact same redirect page source) (Mathematica 11.0 from EU/Hungary) $\endgroup$
    – Gyebro
    Jan 13, 2017 at 15:01
  • $\begingroup$ 404 here. Win10 V11. $\endgroup$
    – Kuba ♦
    Jan 13, 2017 at 15:32
  • $\begingroup$ 404 here as well. MacOS 10.12.2, MMA v.11.0.1 $\endgroup$
    – Cassini
    Jan 13, 2017 at 18:08

3 Answers 3

Reset to default
3
$\begingroup$

Don't know how to automatically avoid that but it seems that everything which is considered "Path" or "Query" part of the url is automatically URLEncoded.

This works on Windows:

URLRead[<|"Scheme" -> "http", "Domain" -> "www.nottingham.ac.uk/~ppzphy/"|>, "Content"]


<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\">
<html>
<head>
<title>Your Page Title</title>
<meta http-equiv=\"REFRESH\" \
content=\"0;url=http://www.nottingham.ac.uk/physics/\"></HEAD>
<BODY>
Redirecting to<br><br>
<a href=\"http://www.nottingham.ac.uk/physics/\">http://www.\
nottingham.ac.uk/physics/</a>
</BODY>
</HTML>
Improve this answer
$\endgroup$
0
4
$\begingroup$

This is an incomplete answer, but it should help. I can reproduce on Windows 7. Using Fiddler it is easy to see that URLFetch issues this request: 

GET /~ppzphy/ HTTP/1.1

while URLRead issues a this one:

GET /%7Eppzphy/ HTTP/1.1

Unfortunately, I didn't find a way to make URLRead work for that address.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, that's helpful. URLFetch["http://www.nottingham.ac.uk/%7Eppzphy/"] returns the 404 error page. $\endgroup$
    – Stephen Powell
    Jan 13, 2017 at 16:20
  • 1
    $\begingroup$ A little off-topic, but opening http://www.nottingham.ac.uk/%7Eppzphy/ in Chrome and IE works correctly, but Firefox gives the 404 page. I think this is a problem with the way the server is set up. $\endgroup$
    – Stephen Powell
    Jan 13, 2017 at 16:31
3
$\begingroup$

It seems like building the URL like so helps:

URLRead[
 HTTPRequest[
  URL["http://www.nottingham.ac.uk"], 
  <|"Path" -> "~ppzphy/"|>
 ]
]

This appears to avoid having URLEncode interfere.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ BTW, I would still report this to [email protected]. At the very least its something they should be aware of. $\endgroup$
    – chuy
    Jan 13, 2017 at 16:26
  • 2
    $\begingroup$ Actually one could well argument that the problem is on the server side which to my understanding should be able to answer both cases ("~" should be escaped according to RFC1738 but the newer RFC2396 classifies it as an "unreserved" character which may or may not be escaped). The real problem on the Mathematica side for me seems to be that it doesn't behave consistently on different plattforms... $\endgroup$
    – Albert Retey
    Jan 13, 2017 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.