Why does importing a URL fail the second time and produce a FetchURL::conopen error?

Question

I have Mathematica code that uses Import[] to download URLs. I noticed that first time I use the Import[] command it works. Next time it fails. I'm using Windows 7, Mathematica 8.0.4.

Here is an example. I start new sesssion of Mathematica, and then type

Import["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]

No problem. Now I issue the same command again on the same URL (or different one)

Import["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]

and now this gives me an error

FetchURL::conopen: The connection to URL http://groups.google.com/group/comp.soft-sys.math.mathematica/about cannot be opened. If the URL is correct, you might need to configure your firewall program, or you might need to set a proxy in the Internet connectivity tab of the Preferences dialog (or by calling SetInternetProxy). For HTTPS connections, you might need to inspect the authenticity of the server's SSL certificate and choose to accept it.

On the same Windows 7 PC, I start MATLAB, and type

data=urlread('http://groups.google.com/group/comp.soft-sys.math.mathematica/about')
data=urlread('http://groups.google.com/group/comp.soft-sys.math.mathematica/about')

and both work. I get the page fine the second time around.

I open the browser, and type the URL, and I can see the page. This is the same PC. No proxy. home PC. Nothing special.

P.S. This code used to work fine with Mathematica on an earlier version (forgot which one now), becuase I used Mathematica to generate this web page and all the Import commands I used are listed above and you can see them. I have list of many URL's that I loop over calling Import[] on them one by one.

Again, the problem is that I now get an error on a URL using Import[] and the problem seems to show up the second time I use the Import[]. Any suggestions how to fix this? Is this a known issue?

update I still have Mathematica 7 and 6 installed. So I tried this on them. And I get the same problem there. This is strange, since this code worked before. I think now it is a google groups/server problem, becuase when I try different domain (i.e. not google groups), I do not get this problem

data = Import["http://www.yahoo.com", "Data"];
data = Import["http://www.yahoo.com", "Data"];

No error in the second time.

I do not understand why this problem (i.e. using the Import command again) gives an error on some URLs and not on others.

update

Thanks for everyone's help and for WReach solution. I have used WReach nice solution to bypass the problem and have just updated the currrent Mathematica code I was using to update the usenet programming languages trends charts. The updated charts are on this page if you are interested in seeing the result.

Answer 1

Watching the network traffic with WireShark reveals that the second and subsequent accesses to the URL fail due to an internal server error.

The initial request looks like this:

GET /group/comp.soft-sys.math.mathematica/about HTTP/1.1
User-agent: Mathematica/8.0.1.0.0 PM/1.3.1
Host: groups.google.com

and yields a successful response with the following headers:

HTTP/1.1 200 OK
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, must-revalidate
Set-Cookie: NID=65=tsm2...ijos;Domain=.google.com;Path=/;Expires=Sun, 05-May-2013 17:41:45 GMT;HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=59d3...eBf7; expires=Mon, 03-Nov-2014 17:41:45 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Date: Sat, 03 Nov 2012 17:41:45 GMT
Server: GWS-GRFE/0.50
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked

The second and subsequent requests include the returned cookies:

GET /group/comp.soft-sys.math.mathematica/about HTTP/1.1
User-agent: Mathematica/8.0.1.0.0 PM/1.3.1
Host: groups.google.com
Cookie: $Version=0; NID=65=tsm2...ijos; $Path=/; $Domain=.google.com
Cookie: $Version=0; PREF=ID=59d3...eBf7; $Path=/; $Domain=.google.com

This elicits a response that indicates a server error:

HTTP/1.1 500 Internal Server Error
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, must-revalidate
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Date: Sat, 03 Nov 2012 17:39:13 GMT
Server: GWS-GRFE/0.50
Content-Length: 1100
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

The body of the response is a generic "server error" message that offers no additional clues:

The server encountered an error and could not complete your request.

If the problem persists, please report your problem and mention this error message and the query that caused it. That...s all we know.

After the Mathematica kernel has been restarted, the Import command once again generates an HTTP request that has no cookies -- which succeeds. So it would appear that the presence of the cookies in the request confuses the server. It is possible that this is an intentional behaviour intended to discourage site access by robots. Perhaps an intermediate cache is struggling with the no-cache, must-revalidate header. Or maybe it is just a programming error in the server.

There is no way to control request cookies through Import or FetchURL. Restarting the kernel is a pretty severe way to work around this problem. A slightly less severe action would be to RestartPacletManager[], but that is still pretty severe.

Workaround

We could just bypass Import altogether and go directly to JLink:

Needs["JLink`"]

httpGet[url_String] :=
    JavaBlock@Module[{http, get}
  , http = JavaNew["org.apache.commons.httpclient.HttpClient"]
  ; get = JavaNew["org.apache.commons.httpclient.methods.GetMethod", url]
  ; http@executeMethod[get]
  ; get@getResponseBodyAsString[]
  ]

Unlike the Paclet Manager, this code uses a fresh HTTPClient each time, so cookies are never retained and this request can be issued successfully any number of times.

httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]

(*
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
...
  <title>comp.soft-sys.math.mathematica | Google Groups</title>
...
*)

httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]

(*
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
....
*)

ImportString can be applied to the result of httpGet to interpret the results:

ImportString[
  httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]
, {"HTML", "Data"}
]

(*
{{{Search,Images,Maps,Play,YouTube,News,Gmail,Documents,
{Calendar,Translate,Mobile,Books,Offers,Wallet,Shopping,Blogger,Reader,Finance,...}},
{Sign in,Help}},
...
*)

ImportString[
  httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]
, {"HTML", "XMLObject"}
]

(*
XMLObject[Document][...]
*)

Answer 2

I actually tried FetchURL[url] and gives same error. Unfortunately FetchURL is not documented. I tried some of the Options[FetchURL], like

FetchURL["http://groups.google.com",   "RequestHeaderFields" ->
{"Cache-control" -> "no-cache",     "Connection" -> "close"},  
"RequestParameters" -> {"x" -> ToString[AbsoluteTime[]]},  
"FileFilters" -> {}]

but couldn't trick the kernel into something else. My (wild) guess is that it is related to the response being cached/blocked by the kernel -including resources!- along with the fact that the above mentioned page contains a broken resource https://groups.google.com/intl/en/images/logos/groups_logo_sm.gif returning 502 (which, by the way, produces the same error on its own)