1
$\begingroup$

I'm using Mathematica10.0.2 on Win7 to write a crawler. Here is a piece of my code:

URLFetch["http://adsabs.harvard.edu/cgi-bin/nph-basic_connect", 
"Method" -> "POST", 
"Parameters" -> {"qsearch" -> 
"Chen Does the F  .   index correctly describe solar EUV flux 
during the deep solar minimum of  ", 
"k" -> "eNpLTClOTCrWy0gsKkssStFLTSkFAEIhBvE="}] 

the key word of "Parameters" is "qsearch", which is got by my httpfox software.

When I search "Chen Does the F . index correctly describe solar EUV flux during the deep solar minimum of " at adsabs.harvard.edu on my chrome and fox, all thing is fine. I got what I want. However, when I run the code above, it throws out 502 error:

 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 <html><head>
 <title>502 Proxy Error</title>
 </head><body>
 <h1>Proxy Error</h1>
 <p>The proxy server received an invalid
 response from an upstream server.<br />
 The proxy server could not handle the request <em><a \
 href="/cgi-bin/nph-basic_connect">POST&nbsp;/cgi-bin/nph-basic_\
 connect</a></em>.<p>
 Reason: <strong>Error reading from remote server</strong></p></p>
 <hr>
 <address>Apache/2.2.3 (CentOS) Server at adsabs.harvard.edu Port \
 80</address>
 </body></html>

What should I do?

When I try

URLFetch["http://adsabs.harvard.edu/cgi-bin/nph-basic_connect", 
"Method" -> "POST", 
"Parameters" -> {"qsearch" -> 
"Chen Does the F  .   index correctly describe solar EUV flux \
during the deep solar minimum of  ", 
"k" -> "eNpLTClOTCrWy0gsKkssStFLTSkFAEIhBvE="}, 
"UserAgent" -> "Chrome", "ConnectTimeout" -> 120, 
"ReadTimeout" -> 120]

It returns :

"Still unable to dial tcp://adsabs.harvard.edu:80 after 3 attempts"

Why would this happen when everything is fine with Chrome?

Does this problem depend on the internet condition? Does the code above actually work for someone?

It seems that I miss a parameter, the code below works well:

URLFetch["http://adsabs.harvard.edu", "Method" -> "POST", 
"Parameters" -> {"qsearch" -> (URLEncode@
  "Chen Does the F  .   index correctly describe solar EUV flux 
during the deep solar minimum of  "), "version" -> "1"}]

However, the parameter "version" is not showed by my httpfox. However, It seems someone could get return even without "Parameters" "version" -> "1"?

$\endgroup$
  • $\begingroup$ Add URLEncode@ in front of the value of qsearch and it'll return a result. When you write an address into the address bar of the browser it automatically encodes it for you. $\endgroup$ – C. E. Aug 8 '16 at 14:43
  • $\begingroup$ I don't know if this helps but URLFetch documentations says that the Method option is without the quotes: Method instead of "Method" $\endgroup$ – Gustavo Delfino Aug 8 '16 at 16:26
  • $\begingroup$ I still get 502 error by "qsearch" -> (URLEncode@ "Chen Does the F . index correctly describe solar EUV flux during the deep solar minimum of ") @C.E. $\endgroup$ – Harry Aug 9 '16 at 4:40
  • $\begingroup$ In the example of Mathematica:URLFetch["http://wolfram.com", "Headers", "Method" -> "HEAD"]. there is quotes on Method. And you can always try whether it would make difference.. $\endgroup$ – Harry Aug 9 '16 at 4:42
  • $\begingroup$ @Harry It doesn't work for me either now, I don't know what happened yesterday. Also I see now that it's a post request... $\endgroup$ – C. E. Aug 9 '16 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.