A response to a previous question about invoking the POST method via Mathematica was most informative in trying to further understand the intricacies of using the POST Method to obtain tabular data from online databases that I then subsequently parse and analyze.
The example given in the link works fine for websites for which there are typically a single name, value pair for each parameter.
However, for one such database, I am stumped as to how to establish the parameters to set for the "RequestParameters" list.
Using the the suggestions from above, I have,
Examine Document Object Model
initialUrl =
"http://www.mncn.csic.es/Menu/Coleccionesydocumentacin/Colecciones/Buscador/\
seccion = 1311&idioma=es_ES.do";
which to establish the Attributes of the Form I use the suggested
Cases[forms[[1]], XMLElement["input", ___], Infinity] // Column
giving
{ {XMLElement[ "input", {"type" -> "hidden", "name" -> "seccion", "value" -> "1323"}, {}]}, {XMLElement[ "input", {"type" -> "hidden", "name" -> "idioma", "value" -> "es_ES"}, {}]}, {XMLElement[ "input", {"type" -> "text", "onclick" -> "if(this.value=='Buscar'){this.value ='';}", "name" -> "valorCampo", "value" -> ""}, {}]} }
No matter what guesses I make to specifiy parameters I provide from the last element of the form, I can not seem to get it to generate a table of results say for the form attached with the above URL to use POST to fill in the "Familia" field with "Scorpaenidae"
parameters = {"seccion" -> "123", "idioma" -> "es_ES",
"onclick" -> "Buscar", "valorCampo" -> "Scorpaenidae"};
results =
Import[initialUrl, "Data", Method -> "POST",
"RequestParameters" -> parameters]
I get a response back for boilerplate from the top of the form, but not the form itself. I am trying to place "Scorpaenidae" into the form field labeled "Familia" as one would do manually, then manipulate the resulting table to drill down for subsequent records in the table. The reason for automatic the procedure is that I could then run such query and associated parsing algorithms to periodically check if new records have been added.
Are there any good references that explain how to infer what the parameters should be based on such syntax for the last XMLElement? I presume this derives from a JavaScript generated form, which perhaps I need to know to do this, but am unsure how to use the result of the Cases statement to construct a proper list of parameters for input into the POST for this specific request.
I've looked on the web and through Mathematica documentation without success.
Likewise, presumably there is a way to "drill down" on the tabular answer to capture successive records when the "tab arrows" of the form that results via manual input are pressed?
Again, any reference as to how parameters to POST methods can be inferred and how such a "drill-downs" could be constructed would be most appreciated. This seems like an area where a good book on web-page mining and scraping using Mathematica would be most useful.
