# URLFetch meet the problem of 502

I'm using Mathematica10.0.2 on Win7 to write a crawler. Here is a piece of my code:

URLFetch["http://adsabs.harvard.edu/cgi-bin/nph-basic_connect",
"Method" -> "POST",
"Parameters" -> {"qsearch" ->
"Chen Does the F  .   index correctly describe solar EUV flux
during the deep solar minimum of  ",
"k" -> "eNpLTClOTCrWy0gsKkssStFLTSkFAEIhBvE="}]


the key word of "Parameters" is "qsearch", which is got by my httpfox software.

When I search "Chen Does the F . index correctly describe solar EUV flux during the deep solar minimum of " at adsabs.harvard.edu on my chrome and fox, all thing is fine. I got what I want. However, when I run the code above, it throws out 502 error:

 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<title>502 Proxy Error</title>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a \
href="/cgi-bin/nph-basic_connect">POST&nbsp;/cgi-bin/nph-basic_\
connect</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
<hr>
</body></html>


What should I do?

When I try

URLFetch["http://adsabs.harvard.edu/cgi-bin/nph-basic_connect",
"Method" -> "POST",
"Parameters" -> {"qsearch" ->
"Chen Does the F  .   index correctly describe solar EUV flux \
during the deep solar minimum of  ",
"k" -> "eNpLTClOTCrWy0gsKkssStFLTSkFAEIhBvE="},
"UserAgent" -> "Chrome", "ConnectTimeout" -> 120,


It returns :

"Still unable to dial tcp://adsabs.harvard.edu:80 after 3 attempts"


Why would this happen when everything is fine with Chrome?

Does this problem depend on the internet condition? Does the code above actually work for someone?

It seems that I miss a parameter, the code below works well:

URLFetch["http://adsabs.harvard.edu", "Method" -> "POST",
"Parameters" -> {"qsearch" -> (URLEncode@
"Chen Does the F  .   index correctly describe solar EUV flux
during the deep solar minimum of  "), "version" -> "1"}]


However, the parameter "version" is not showed by my httpfox. However, It seems someone could get return even without "Parameters" "version" -> "1"?

• Add URLEncode@ in front of the value of qsearch and it'll return a result. When you write an address into the address bar of the browser it automatically encodes it for you. – C. E. Aug 8 '16 at 14:43
• I don't know if this helps but URLFetch documentations says that the Method option is without the quotes: Method instead of "Method" – Gustavo Delfino Aug 8 '16 at 16:26
• I still get 502 error by "qsearch" -> (URLEncode@ "Chen Does the F . index correctly describe solar EUV flux during the deep solar minimum of ") @C.E. – Harry Aug 9 '16 at 4:40
• In the example of Mathematica:URLFetch["http://wolfram.com", "Headers", "Method" -> "HEAD"]. there is quotes on Method. And you can always try whether it would make difference.. – Harry Aug 9 '16 at 4:42
• @Harry It doesn't work for me either now, I don't know what happened yesterday. Also I see now that it's a post request... – C. E. Aug 9 '16 at 9:49