A response to a previous question about invoking the POST method via Mathematica was most informative in trying to further understand the intricacies of using the POST Method to obtain tabular data from online databases that I then subsequently parse and analyze.

The example given in the link works fine for websites for which there are typically a single name, value pair for each parameter.

However, for one such database, I am stumped as to how to establish the parameters to set for the "RequestParameters" list.

Using the the suggestions from above, I have,

Examine Document Object Model
initialUrl = 
seccion = 1311&idioma=es_ES.do";

which to establish the Attributes of the Form I use the suggested

Cases[forms[[1]], XMLElement["input", ___], Infinity] // Column


{ {XMLElement[ "input", {"type" -> "hidden", "name" -> "seccion", "value" -> "1323"}, {}]}, {XMLElement[ "input", {"type" -> "hidden", "name" -> "idioma", "value" -> "es_ES"}, {}]}, {XMLElement[ "input", {"type" -> "text", "onclick" -> "if(this.value=='Buscar'){this.value ='';}", "name" -> "valorCampo", "value" -> ""}, {}]} }

No matter what guesses I make to specifiy parameters I provide from the last element of the form, I can not seem to get it to generate a table of results say for the form attached with the above URL to use POST to fill in the "Familia" field with "Scorpaenidae"

parameters = {"seccion" -> "123", "idioma" -> "es_ES", 
"onclick" -> "Buscar", "valorCampo" -> "Scorpaenidae"};

results = 
Import[initialUrl, "Data", Method -> "POST", 
"RequestParameters" -> parameters]

I get a response back for boilerplate from the top of the form, but not the form itself. I am trying to place "Scorpaenidae" into the form field labeled "Familia" as one would do manually, then manipulate the resulting table to drill down for subsequent records in the table. The reason for automatic the procedure is that I could then run such query and associated parsing algorithms to periodically check if new records have been added.

Are there any good references that explain how to infer what the parameters should be based on such syntax for the last XMLElement? I presume this derives from a JavaScript generated form, which perhaps I need to know to do this, but am unsure how to use the result of the Cases statement to construct a proper list of parameters for input into the POST for this specific request.

I've looked on the web and through Mathematica documentation without success.

Likewise, presumably there is a way to "drill down" on the tabular answer to capture successive records when the "tab arrows" of the form that results via manual input are pressed?

Again, any reference as to how parameters to POST methods can be inferred and how such a "drill-downs" could be constructed would be most appreciated. This seems like an area where a good book on web-page mining and scraping using Mathematica would be most useful.

  • $\begingroup$ Maybe you need to send a cookie with your request. $\endgroup$ Nov 3, 2015 at 22:17
  • $\begingroup$ Not sure how this might be done. Presently, just trying to understand more about how Mathematica interacts with the server via the various IP levels, headers, responses, etc. Do you have a reference(s) or more specifics as to what might have prompted your suggestion? $\endgroup$ Nov 4, 2015 at 22:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.