30
$\begingroup$

I wanted to input some data on this page through Mathematica. Then I would press the "buscar" button also through Mathematica coding and get the results on Mathematica. The most similar post I've found was this one.

I'm trying to import through the Import[] function:

Import["http://www.fundamentus.com.br/buscaavancada.php", "Data"]

When I do that, it shows this:

{{{{"Página inicial", "Investimento consciente", 
"Entre em contato"}, {"Detalhes", {"Balanço patrimonial", 
 "Demonstrativos de resultados", 
 "Indicadores fundamentalistas"}, {"Balanços em Excel", 
 "Proventos"}, 
"Histórico de cotações"}}, {{"Preencha apenas os campos que serão \
usados na pesquisa", "Use . (ponto) como separador decimal", 
"Utilize o formato decimal para representar porcentagens. Ex: \
Para 15% digite 0.15"}, {{"? P/L (Preço sobre lucro)", 
 "min max"}, {"? P/VP (Preço sobre Valor Patrimonial)", 
 "min max"}, {"? PSR (Preço sobre Receita Líquida)", 
 "min max"} [...]

And from here I'm stuck. Any tips?

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ so you want to populate the input fields and import the result? What you need to do is have a look at the page source and figure out what query strings are being sent when you populate the input fields. Then you can create the direct URL that is created when you press "buscar" $\endgroup$
    – Mike Honeychurch
    Feb 26, 2012 at 23:35
  • 1
    $\begingroup$ If a POST request is used, an alternative to @Mike's suggestion would be to install some browser plugin which intercepts requests, to read off the actual request sent. For Firefox, e.g., there is Tamper Data. $\endgroup$
    – Leonid Shifrin
    Feb 26, 2012 at 23:48
  • 5
    $\begingroup$ Import has experimental support for POST requests. Please see here. As Leonid suggested, use some debugging tool to figure out what the precise request is, then use either the experimental POST support in Import, or if that doesn't work (e.g. multi-part requests) then you can try calling command line tools such as curl or wget from Mathematica. $\endgroup$
    – Szabolcs
    Feb 27, 2012 at 8:42
  • $\begingroup$ I'm reading on wikipedia and there are some request methods: (OPTIONS · GET · HEAD · POST · PUT · DELETE · TRACE · CONNECT · PATCH) Only GET and POST are useful for this kind of operation? $\endgroup$
    – Red Banana
    Feb 28, 2012 at 4:25
  • 1
    $\begingroup$ Yes, usually you only need to use one of these two. GET is supported by Import officially, but your site uses POST. GET means that everything is put in the URL, so it's easy. It is what Mike is referring to. $\endgroup$
    – Szabolcs
    Feb 28, 2012 at 17:28

3 Answers 3

Reset to default
37
$\begingroup$

The first thing we need to do is to determine how the initial page assembles the parameters and transmits the request to the server. One way to do this would be to open the initial page using the developer tools in the web browser. But since this is a Mathematica forum, let's try to use the tools it makes available to us.

We could load the page text and then try to extract the information we need using string manipulation functions. However, this can get tricky as we must account for line breaks in inconvenient locations, decode HTML entities, and so on. Instead, we will examine the page's Document Object Model (DOM). In Mathematica, the DOM is accessed by importing the page using "XMLObject" format:

$initialUrl = "http://www.fundamentus.com.br/buscaavancada.php";
$dom = Import[$initialUrl, "XMLObject"];

Fewer and fewer pages these days are using simple HTML forms to send requests to the server -- let's see if this page contains any FORM elements:

$forms = Cases[$dom, XMLElement["form", ___], Infinity];
Length @ $forms

2

We are in luck. Let's look at the attributes of the forms:

Cases[$forms, XMLElement[_, attrs_, _] :> attrs]

{
 {enctype->application/x-www-form-urlencoded,method->get,
    class->busca,action->detalhes.php},
 {enctype->application/x-www-form-urlencoded,method->post,
    class->avancada,name->formbusca,action->resultado.php}
}

The first form ("detalhes") uses HTTP GET to get its results. The second ("resultado") uses POST. Resultado sounds promising. Let's extract the input elements for that form:

Cases[$forms[[2]], XMLElement["input", ___], Infinity] // Column

XMLElement[input,{type->text,name->pl_min},{}]
XMLElement[input,{type->text,name->pl_max},{}]
XMLElement[input,{type->text,name->pvp_min},{}]
... lines omitted ...
XMLElement[input,{type->text,name->roe_min},{}]
XMLElement[input,{type->text,name->roe_max},{}]
XMLElement[input,{type->text,name->liq_min},{}]
XMLElement[input,{type->text,name->liq_max},{}]
... lines omitted ...

Yes, this looks like the form that we are interested in. Let's assemble the components of a request:

$resultUrl = StringReplace[$initialUrl, "buscaavancada.php" -> "resultado.php"]

http://www.fundamentus.com.br/resultado.php

$parameters = {
  "roe_min" -> "0.1"
, "liq_min" -> "500000"
, "liq_max" -> "800000"
};

... and transmit the request using HTTP POST:

$results = Import[
  $resultUrl
, "Data"
, "RequestMethod" -> "POST"
, "RequestParameters" -> $parameters
]

{{{{Página inicial,Investimento consciente,Entre em contato},{Detalhes,{Balanço patrimonial,Demonstrativos de resultados,Indicadores fundamentalistas},{Balanços em Excel,Proventos},Histórico de cotações}},{{Papel,Cotação,P/L,P/VP,PSR,Div.Yield,P/Ativo,P/Cap.Giro,P/EBIT,P/Ativ Circ.Liq,EV/EBIT,Mrg Ebit,Mrg. Líq.,Liq. Corr.,ROIC,ROE,Liq.2meses,Patrim. Líq,Dív.Brut/ Patrim.,Cresc. Rec.5a},{{PRTX3,2,72,-38,34,-255,71,905,562,0,00%,1,977,-5,52,-63,14,-2,42,-72,49,-1.434,22%,-2.361,99%,0,35,-3,96%,666,96%,537.768,00,-10.557.000,00,-59,73,0,00%}
... and more ...

This time we have imported using the "Data" format which let's Mathematica do all the hard work of extracting the HTML TABLE elements out of the web page.

At this point, we have successfully imported all of the data into Mathematica. We can now use the usual Mathematica tools to extract and reformat those parts that interest us. After a bit of experimentation, we can see that the interesting data is the the second element of the first row:

$interesting = $results[[1, 2]];
$interesting // TableForm

raw data table

We can extract the property names:

$propertyNames = $interesting[[1, 2;;]]

{Cotação,P/L,P/VP,PSR,Div.Yield,P/Ativo,P/Cap.Giro,P/EBIT,P/Ativ Circ.Liq,EV/EBIT,Mrg Ebit,Mrg. Líq.,Liq. Corr.,ROIC,ROE,Liq.2meses,Patrim. Líq,Dív.Brut/ Patrim.,Cresc. Rec.5a}

... and the ticker symbols:

$symbols = $interesting[[2, All, 1]]

{PRTX3,BRTO3,FHER3,PINE4}

... and the data itself:

$data = $interesting[[2, All, 2;;]]

{{2,72,-38,34,-255,71,905,562,0,00%,1,977,-5,52,-63,14,-2,42,-72,49,-1.434,22%,-2.361,99%,0,35,-3,96%,666,96%,537.768,00,-10.557.000,00,-59,73,0,00%},{12,15,3,87,0,68,0,771,2,46%,0,256,4,35,2,62,-0,85,2,97,29,44%,19,90%,1,22,12,50%,17,68%,750.626,00,10.699.600.000,00,0,53,-3,53%},{12,25,4,48,1,38,0,135,0,00%,0,201,-12,78,1,64,-2,14,3,72,8,25%,3,02%,0,98,22,47%,30,87%,686.507,00,429.309.000,00,2,52,7,74%},{12,39,7,58,1,21,0,000,7,39%,0,000,0,00,0,00,0,00,0,00,0,00%,0,00%,0,00,0,00%,15,91%,509.960,00,1.015.080.000,00,0,00,-10,02%}}

Since the numbers and percentages were not in a format that Mathematica recognizes, they were imported as strings. We need to convert those strings into Mathematica syntax so that we can parse them:

parse[s_String] /; StringMatchQ[s, __~~"%"] :=
  parse[StringDrop[s, -1]] / 100

parse[s_String] /; StringMatchQ[s, (DigitCharacter|"-"|","|".")..] :=
  ToExpression[StringReplace[s, {"," -> ".", "." -> ""}]]

parse[s_] := s

$data2 = $data /. s_String :> parse[s]

{{2.72,-38.34,-255.71,905.562,0.00,1.977,-5.52,-63.14,-2.42,-72.49,-1434.22,-2361.99,0.35,-3.96,666.96,537768.00,-10557000.00,-59.73,0.00},{12.15,3.87,0.68,0.771,2.46,0.256,4.35,2.62,-0.85,2.97,29.44,19.90,1.22,12.50,17.68,750626.00,10699600000.00,0.53,-3.53},{12.25,4.48,1.38,0.135,0.00,0.201,-12.78,1.64,-2.14,3.72,8.25,3.02,0.98,22.47,30.87,686507.00,429309000.00,2.52,7.74},{12.39,7.58,1.21,0.000,7.39,0.000,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,15.91,509960.00,1015080000.00,0.00,-10.02}}

Note that percentages were converted into fractions.

We can now display the data or manipulate it as we see fit, for example:

TableForm[Transpose @ $data2, TableHeadings -> {$propertyNames, $symbols}]

formatted data table

Improve this answer
$\endgroup$
10
  • 1
    $\begingroup$ I'm slightly confused about why you're using variable names with dollar signs in front of them. Aren't such variable names disrecommended by Mathematica? Otherwise, this is a magnificent answer. $\endgroup$
    – Reb.Cabin
    Mar 3, 2012 at 17:28
  • 2
    $\begingroup$ @Reb.Cabin I use dollar signs for ad hoc global variables because I have been bitten too many times by accidentally assigning an own-value to a symbol that I happen to use, e.g., in a symbolic equation. So I am in the habit of writing $x instead of x. I stick to that convention even for longer symbols like $initialUrl. Just like the regular symbol convention, Mathematica's predefined global symbols all start with a capital letter after the $. My ad hoc symbols always start with a lower case letter, so there is no conflict. $\endgroup$
    – WReach
    Mar 3, 2012 at 17:35
  • 1
    $\begingroup$ @WReach: thanks a lot, this is wonderful. One question: is it possible to navigate many pages inside a websiste that needs a log-in if I log myself with a POST like you did? My real question is: does the internal web engine of Mathematica handle cookies? Or it can just send a POST request but nothing more? $\endgroup$
    – Francesco
    Mar 5, 2012 at 17:13
  • 1
    $\begingroup$ @Francesco Mathematica ships with many extra packages in the installation directory. Many are undocumented but are presumably used internally by Mathematica itself. However, one can learn a lot by looking at those packages. URLTools, for example, can be found here: SystemOpen@ FileNameJoin[{$InstallationDirectory, "AddOns", "ExtraPackages", "Utilities"}]. $\endgroup$
    – WReach
    Mar 8, 2012 at 19:47
  • 1
    $\begingroup$ @RedBanana The use of XML here is incidental. It just so happens that WL uses the same model for not only XML documents but also HMTL documents like the web page under discussion. As for ontology, it is still the common case that the ontology of a web page is ad hoc and that we must reverse engineer it by inspection. There is a movement to use formal ontologies in web pages, but it has yet to be widely adopted. See, for example, the ld+json script block in the source of any IMDB movie page which uses the schema.org ontology. $\endgroup$
    – WReach
    Mar 16 at 16:05
5
$\begingroup$

I stumbled on this and thought I'd share a solution to a recent problem. I had a lot of HTML to parse and wanted to take advantage of jQuery selectors in Mathematica. I used .NETLink. The code illustrates the basics of creating a .NET WebBrowser and interacting with it.

Create a .NET Form with a WebBrowser and return handles to both:

Needs["NETLink`"];
browserForm := Module[{fm, wb},
 fm = NETNew["System.Windows.Forms.Form"];
 fm@Show[];(*or Hide[]*)
 wb = NETNew["System.Windows.Forms.WebBrowser"];
 wb@Parent = fm;
 LoadNETType["System.Windows.Forms.DockStyle"];
 wb@Dock = DockStyle`Fill;
 wb@ScriptErrorsSuppressed = True;
 {fm, wb}];

This function executes JavaScript on the page and reads the result. It assumes jQuery is already loaded but you could use the same technique to load it first if that is not the case.

Options[executeScript] = {
 "Stringify" -> False,
 "Map" -> False};

executeScript[b_(*browser*), script_String, OptionsPattern[]] := NETBlock[Module[{
  scr = script,
  sfy = OptionValue@"Stringify",
  mp = OptionValue@"Map",
  pw = b[Document][DomDocument][parentWindow]},
 If[
  b[Document][GetElementById["mathematicaResult"]] === Null,
  pw[execScript[
    "$('body').append('<div id=\"mathematicaResult\"></div>')"]],
      pw[execScript["$('#mathematicaResult').html('')"]]];
 If[Head[mp] == String, sfy = True; 
  scr = "$('" <> scr <> "').map(function(){" <> mp <> "})"];
     If[sfy, scr = "JSON.stringify(" <> scr <> ")"];
     pw[execScript[
       "$('#mathematicaResult').html('<div \
   id=\"mathematicaResult\">'+" <> scr <> "+'</div>');"]];
 b[Document][GetElementById["mathematicaResult"]][InnerText]]];

Then I was able to write one-liners like

dates = executeScript[b, "td:nth-of-type(12n+10)", "Map" -> "return $(this).html()"];

Depending on the page, you could probably manage most of your objective (filling forms, clicking buttons, reading data) quite easily using only the WebBrowser class (as opposed to running scripts). It would be worth your time to scan that documentation and also the docs on HtmlDocument and HtmlElement (and the basics of NETLink, the key points of which you can see above).

Improve this answer
$\endgroup$
2
$\begingroup$

This is most informative as I have been trying to understand the intricacies of using the POST Method via Mathematica to obtain tabular data that I then subsequently parse.

The example works fine for websites for which there are typically a single name, value pair for each parameter.

However, for one, I am stumped as to how to establish the parameters to set for the "RequestParameters" list.

Using the above, I have,

Examine Document Object Model
initialUrl = 
       "http://www.mncn.csic.es/Menu/Coleccionesydocumentacin/Colecciones/Buscador/\
seccion=1311&idioma=es_ES.do";

which to establish the Attributes of the Form I use the suggested

Cases[forms[[1]], XMLElement["input", ___], Infinity] // Column

giving

{ {XMLElement[ "input", {"type" -> "hidden", "name" -> "seccion", "value" -> "1323"}, {}]}, {XMLElement[ "input", {"type" -> "hidden", "name" -> "idioma", "value" -> "es_ES"}, {}]}, {XMLElement[ "input", {"type" -> "text", "onclick" -> "if(this.value=='Buscar'){this.value ='';}", "name" -> "valorCampo", "value" -> ""}, {}]} }

No matter what combination of parameters I provide from the last element of the form, I can not seem to get it to generate a table of results say for a form filling the "Familia" field with "Scorpaenidae"

parameters = {"seccion" -> "123", "idioma" -> "es_ES", 
"onclick" -> "Buscar", "valorCampo" -> "Scorpaenidae"};

results = 
Import[initialUrl, "Data", Method -> "POST", 
"RequestParameters" -> parameters]

I get information from the top of the form, but not the form itself, nor the tabular data I would get if I entered "Scorpaenidae" into the "Familia" field.

Is there a good reference that explains how to infer what parameters should be based on such syntax for such an XMLElement? I presume this derives from JavaScript but am unsure how to manipulate the result of the Cases statement to construct a proper list of parameters for input into the POST request.

I've looked on the web and through Mathematica documentation without success.

Likewise, presumably there is a way to "drill down" on the tabular answer to capture successive records when the tab arrows of the form that results via manual input are pressed. Again any reference as to how parameters to POST methods can be inferred and how such a "drill-down" could be constructed would be most appreciated. This seems like an area where a good book on Webpage mining and scraping would be most useful.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Answers are meant for, well, answers. If you're asking a new question, ask it here and link to this question if needed. $\endgroup$
    – J. M.'s eventual burnout ♦
    Nov 3, 2015 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.