3
$\begingroup$

I'm trying to download data from website automatically using Mathematica, and URLFetch seems like it should be suited to the task. I'm interested in its "passing data to the server" option.

The example from ref/URLFetch in document:

URLFetch["search.wolfram.com", 
"Parameters" -> {"query" -> "Mathematica HTTPClient"}]

How can I get the key word "query" in "Parameters" at the first time?

I'm struggling on this website. How can I figure out what kind of parameters that I should provide in "Parameters" -> {unknownParam} I've tried something like:

  URLFetch["http://madrigal.iggcas.ac.cn/cgi-bin/madrigal/gSimpleUIAccessData.py", "All"]

but I still can't find any info on what to specify as the Parameters keyword.

Here's a screenshot of the website I'm trying to work with:

enter image description here

$\endgroup$
  • $\begingroup$ Related question: How to manipulate web pages on Mathematica?. $\endgroup$ – WReach Oct 12 '14 at 14:04
  • $\begingroup$ I believe, the keyword "query" is related to the name in the <input type="text" name="query"... snippet of code in the document $\endgroup$ – LLlAMnYP Aug 19 '15 at 11:32
4
$\begingroup$

LLIAMnYP is correct, so this question isn't really about Mathematica. Nonetheless I will provide a very short intro to how this works. If you look at the source code, which you can do easily in any of the major web browsers, you can see that for your website of it is this:

screenshot of code

This screenshot is of how the HTML code looks in Google Chrome's developer tool.

There are four lines that matter to us:

  • <form method="get" action="logInUser.py" enctype="...">
  • <input name="user_fullname" size="50">
  • <input name="user_email" size="50">
  • <input name="user_affiliation" size="50">

First of all, the parameters that you are talking about cannot be known a priori. They are just variable names, and they are just as arbitrary as variable names in mathematics. They were chosen by the developer who wrote the script that lives on the website's server. Figuring out these parameters, then, is a form of reverse engineering.

The first line tells us that when this form is submitted the web browser should send a GET request to the Python script logInUser.py. This is the script that expects certain parameter names, and even though we don't know what's in this script we know that sending this script the parameters user_fullname, user_email and user_affiliation works - because that's what the browser is doing.

Using this information that we've extracted, we can figure out that URLFetch should be used in this manner:

URLFetch[
"http://madrigal.iggcas.ac.cn/cgi-bin/madrigal/logInUser.py",
Method -> "GET",
Parameters -> {
"user_fullname" -> "my_name",
"user_email" -> "my_email",
"user_affiliation" -> "my_affiliation"
}]

You can go even further to make your request look more like the request that the browser is sending to the server. Notably, UserAgent is a piece of information that is sent along with the request to identify what software made the request. This can be changed to match you browser, so the server thinks the request came from that browser instead of Mathematica.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.