How can I determine `Parameters` for use in `URLFetch`?

Question

I'm trying to download data from website automatically using Mathematica, and URLFetch seems like it should be suited to the task. I'm interested in its "passing data to the server" option.

The example from ref/URLFetch in document:

URLFetch["search.wolfram.com", 
"Parameters" -> {"query" -> "Mathematica HTTPClient"}]

How can I get the key word "query" in "Parameters" at the first time?

I'm struggling on this website. How can I figure out what kind of parameters that I should provide in "Parameters" -> {unknownParam} I've tried something like:

  URLFetch["http://madrigal.iggcas.ac.cn/cgi-bin/madrigal/gSimpleUIAccessData.py", "All"]

but I still can't find any info on what to specify as the Parameters keyword.

Here's a screenshot of the website I'm trying to work with:

Answer 1

LLIAMnYP is correct, so this question isn't really about Mathematica. Nonetheless I will provide a very short intro to how this works. If you look at the source code, which you can do easily in any of the major web browsers, you can see that for your website of it is this:

This screenshot is of how the HTML code looks in Google Chrome's developer tool.

There are four lines that matter to us:

• <form method="get" action="logInUser.py" enctype="...">
• <input name="user_fullname" size="50">
• <input name="user_email" size="50">
• <input name="user_affiliation" size="50">

First of all, the parameters that you are talking about cannot be known a priori. They are just variable names, and they are just as arbitrary as variable names in mathematics. They were chosen by the developer who wrote the script that lives on the website's server. Figuring out these parameters, then, is a form of reverse engineering.

The first line tells us that when this form is submitted the web browser should send a GET request to the Python script logInUser.py. This is the script that expects certain parameter names, and even though we don't know what's in this script we know that sending this script the parameters user_fullname, user_email and user_affiliation works - because that's what the browser is doing.

Using this information that we've extracted, we can figure out that URLFetch should be used in this manner:

URLFetch[
"http://madrigal.iggcas.ac.cn/cgi-bin/madrigal/logInUser.py",
Method -> "GET",
Parameters -> {
"user_fullname" -> "my_name",
"user_email" -> "my_email",
"user_affiliation" -> "my_affiliation"
}]

You can go even further to make your request look more like the request that the browser is sending to the server. Notably, UserAgent is a piece of information that is sent along with the request to identify what software made the request. This can be changed to match you browser, so the server thinks the request came from that browser instead of Mathematica.