37
$\begingroup$

Is there a Mathematica API for the functions.wolfram site?
If there's not, has anyone implemented a web scraper for it?

For example it would be nice to be able to access http://functions.wolfram.com/07.23.17.0081.01 from within Mathematica by using something like

In[1]:= FunctionsWolfram["07.23.17.0081.01", InputForm]
Out[1]= Hypergeometric2F1[a,b,a+b-1/2,z]
        == Hypergeometric2F1[2 a-1,2 b-1,a+b-1/2,1/2 (1-Sqrt[1-z])]/Sqrt[1-z]

In[2]:= FunctionsWolfram["07.23.17.0081.01", RuleForm]
Out[2]= HoldPattern[Hypergeometric2F1[a_,b_,a_+b_-1/2,z_]] 
        :> Hypergeometric2F1[2 a-1,2 b-1,a+b-1/2,1/2 (1-Sqrt[1-z])]/Sqrt[1-z]

In[3]:= FunctionsWolfram["07.23.17.0081.01", TraditionalForm]
Out[3]=

Image_from_http://functions.wolfram.com/07.23.17.0081.01

$\endgroup$
0

4 Answers 4

19
$\begingroup$

Here is a shameless plug for my HTML parser posted here. The code is a bit long to reproduce here, the only change to it I'd do is to replace the function processPosList with this code:

processPosList::unmatched = "Unmatched lists `1` enountered!";
processPosList[{openlist_List, closelist_List}] := 
  Module[{opengroup, closegroup, poslist}, 
  {opengroup, closegroup} = groupPositions /@ {openlist, closelist};
   poslist = Transpose[Transpose[Sort[#]] & /@ {opengroup, closegroup}];
   If[UnsameQ @@ poslist[[1]], Return[(Message[
       processPosList::unmatched , {openlist, closelist}]; {})], 
   poslist = Transpose[{poslist[[1, 1]], Transpose /@ Transpose[poslist[[2]]]}]]];

which will issue a message when some parts can not be parsed instead of printing the details (as the original code does). I must warn that my parser for some reason can not fully parse the Wolfram Functions pages (either they are ill-formed or my parser contains bugs), but it will parse enough for our purposes. Here is a simple web-scraper based on it and on a few observations about the typical format of the page:

Clear[getForms];
getForms[url_String] := 
 Quiet@ Cases[postProcess@parseText[Import[url, "Text"]],
     pContainer[attribContainer[" class='CitationInfo'"], x__String] :> 
        StringJoin@x, Infinity] //. 
       x_String :>  StringReplace[ x, {""" | "quot;" :> "\"", "&" :> "", 
             "&lt;" | "&lt" :> "<", "&gt;" | "&gt" :> ">", "\n" :> " "}];


Clear[formsOk, getInputForm, getStandardForm, getRuleForm];
formsOk[forms_] := Length[forms] == 5;
getInputForm[forms_?formsOk] := ToExpression[forms[[1]], InputForm];
getStandardForm[forms_?formsOk] := ToExpression[First@ToExpression[forms[[2]]], StandardForm];
getRuleForm[forms_?formsOk] := ToExpression[First@ToExpression[forms[[4]]]];
getInputForm[__] = getStandardForm[__] = getRuleForm[__] = $Failed;

I can not say how fragile this is, probably rather fragile. Here is an example of use:

In[277]:= 
forms = getForms["http://functions.wolfram.com/07.23.17.0084.01"];
Through[{getInputForm,getStandardForm,getRuleForm}[forms]]

Out[278]= {Hypergeometric2F1[a,b,-(1/2)+a+b,z]==((Sqrt[1-z]-Sqrt[-z])^(1-2 a) 
  Hypergeometric2F1[-1+2 a,-1+a+b,-2+2 a+2 b,2 z+2 Sqrt[-z+z^2]])/Sqrt[1-z]/;Re[z]>1/2,
  Hypergeometric2F1[a,b,-(1/2)+a+b,z]==((Sqrt[1-z]-Sqrt[-z])^(1-2 a) 
  Hypergeometric2F1[-1+2 a,-1+a+b,-2+2 a+2 b,2 z+2 Sqrt[-z+z^2]])/Sqrt[1-z]/;Re[z]>1/2,
  HoldPattern[Hypergeometric2F1[a_,b_,a_+b_-1/2,z_]]:>((Sqrt[1-z]-Sqrt[-z])^(1-2 a) 
  Hypergeometric2F1[2 a-1,a+b-1,2 a+2 b-2,2 Sqrt[z^2-z]+2 z])/Sqrt[1-z]/;Re[z]/2}

I tested on about 10 different formulas, and this worked fine, but of course this is not an extensive test, so most likely this will not always work.

$\endgroup$
2
  • $\begingroup$ Thanks Leonid! I'll have a closer look at this tomorrow. I wouldn't be too concerned about its fragility - it's probably a lot more robust than my solution... $\endgroup$
    – Simon
    May 31, 2011 at 11:57
  • $\begingroup$ @Simon For your solution, the heavy lifting is done with Import, which does a pretty good job of importing in default format (HTML here). My solution parses HTML and looks at certain pattern in the parsed document. For the case at hand, your solution seems much more robust and adequate, so I am tempted to delete mine - perhaps will give it a day or two... $\endgroup$ May 31, 2011 at 12:08
14
$\begingroup$

The site is not terribly conducive to scraping as the HTML is "noisy" and looks like WRI might change the format at the drop of a hat. Throwing caution to the wind...

scrapeWolframFunction[id_] :=
  Import["http://functions.wolfram.com/" ~~ id, "XMLObject"] //
  Cases[
    #
  , XMLElement["p", {___, "class" -> "CitationInfo", ___}, body_] :> body
  , Infinity
  ] & //
  ToExpression[#[[1, 1]], InputForm, HoldForm] &

The function assumes that the first CitationInfo paragraph contains the InputForm. This assumption appears to hold true for the moment.

Sample use:

In[24]:= scrapeWolframFunction["07.23.17.0081.01"]
Out[24]= Hypergeometric2F1[a,b,a+b-1/2,z]==Hypergeometric2F1[2 a-1,2 b-1,a+b-1/2,1/2 (1-Sqrt[1-z])]/Sqrt[1-z]

In[25]:= scrapeWolframFunction["01.06.02.0001.01"]
Out[25]= Sin[z]==(E^(I z)-E^(-I z))/(2 I)

In[26]:= scrapeWolframFunction["06.25.27.0004.01"]
Out[26]= Erf[z]==(1+I) (FresnelC[((1-I) z)/Sqrt[\[Pi]]]-I FresnelS[((1-I) z)/Sqrt[\[Pi]]])
$\endgroup$
1
  • 4
    $\begingroup$ Mathematica's built-in HTML parser is TagSoup which is very good at handling HTML as it is found in the wild. It is worth the effort to persevere through XMLObject if one has any need to parse either HTML or XML. $\endgroup$
    – WReach
    Jun 2, 2011 at 3:42
9
$\begingroup$

Here's a quick scraper that I built (after asking the question). So far it has minimal error checking. Also note that InputForm, StandardForm and MathMLForm should all yield the same expressions.

FunctionsWolframIDQ[id_String]:=StringMatchQ[id,
  RegularExpression["\\d\\d\\.\\d\\d\\.\\d\\d\\.\\d\\d\\d\\d\\.\\d\\d"]]
FWIDQ=FunctionsWolframIDQ;

FunctionsWolfram[id_String?FWIDQ]:=FunctionsWolfram[id,InputForm]

FunctionsWolfram[id_String?FWIDQ,All] := FunctionsWolfram[id,All] = 
  Module[{imp=Import["http://functions.wolfram.com/"<>id]}, StringSplit[imp,
    "Input Form"|"Standard Form"|"MathML Form"|"Rule Form"|"Date Added"]]

FunctionsWolfram[id_String?FWIDQ,InputForm] := 
  ToExpression@StringTrim@FunctionsWolfram[id,All][[2]]

FunctionsWolfram[id_String?FWIDQ,StandardForm] := 
  ToExpression@First@ToExpression@StringTrim@FunctionsWolfram[id,All][[3]]

FunctionsWolfram[id_String?FWIDQ,MathMLForm] :=
  ToExpression[StringTrim@FunctionsWolfram[id,All][[4]],MathMLForm]

FunctionsWolfram[id_String?FWIDQ,RuleForm] := 
  ToExpression@First@ToExpression@StringTrim@FunctionsWolfram[id,All][[5]]

FunctionsWolfram[id_String?FWIDQ, TraditionalForm] := 
  TraditionalForm[FunctionsWolfram[id, InputForm]]

It works as advertised in the question.

$\endgroup$
6
  • $\begingroup$ This was just a really quick solution that just reads a functions site in as a long string and then splits it up and spits out the appropriate parts. I left the answer "deleted" until I was ready to call it a night and not check SO until morning! $\endgroup$
    – Simon
    May 31, 2011 at 11:56
  • $\begingroup$ Nice. I was already doing messy pattern matching in the html code read in using Import[...,"Text"]: StringReplace[text, ___ ~~ "<font class='CitationLabel' size='-1'>Input Form</font>" ~~ Shortest[_] ~~ "<p class='CitationInfo'>" ~~ Shortest[inputForm ~~ "</p>"] ~~ ___ :> inputForm]. This is much cleaner $\endgroup$ May 31, 2011 at 12:25
  • 1
    $\begingroup$ @Sjoerd Re your comment. Have you read this excellent post: stackoverflow.com/questions/1732348/…? $\endgroup$ Jun 3, 2011 at 3:47
  • 1
    $\begingroup$ It seems to me this great function would gain a lot if it could take WildCards, such as FunctionWolfram["HyperGeometric2F1"] so that it would returns all identities related to HyperGeometric2F1 funtions. Once the index of such entries identified one could imagine using something like Table[FunctionsWolfram[ "07.23.17." <> IntegerString[i, 10, 4] <> ".01"], {i, 4}]? The purpose of this would be to produce transformation rules that the user could map onto given mathematica results to suit his/her need. $\endgroup$
    – chris
    Apr 28, 2012 at 18:54
  • $\begingroup$ @chris: That does sound great! The functions site used to have and advanced search form that used standard Mathematica patterns... but I can't find it on the site anymore. Feel free to develop something more useful than what has been posted here and I'll accept your answer (as I couldn't choose between the three answers already posted). $\endgroup$
    – Simon
    Apr 29, 2012 at 4:12
6
$\begingroup$

The new MathematicalFunctionData[] function seems to know a lot of the identities that are in the Wolfram Functions site. Unfortunately, I haven't quite figured out how to have it recognize a permalink on the Wolfram Functions site, so here's an alternative way to look up the formula featured in the OP:

idList = MathematicalFunctionData[Hypergeometric2F1, "FunctionalEquations", 
                                  "IncludedSubexpressions" -> {HoldPattern[Sqrt[_]]}]

and looking at the list thus returned reveals that the first one is the desired identity:

id = First[idList]
   Function[{a, b, z},
            Inactivate[Hypergeometric2F1[a, b, a + b - 1/2, z] ==
            (1/Sqrt[1 - z]) Hypergeometric2F1[2 a - 1, 2 b - 1, a + b - 1/2,
                                              (1 - Sqrt[1 - z])/2]]]

(N.B. I had changed the formal symbols that are actually in the output to normal letters for clarity.)

One can now do something like

id[a, b, z] // Activate // TraditionalForm

to show the desired identity in the traditional notation, or

id[-1/4, 1/4, z] // Activate
   Hypergeometric2F1[-1/4, 1/4, -1/2, z] ==
   (1 + (-1 + Sqrt[1 - z])/2)^(3/2)/Sqrt[1 - z]

to display a special case.

The documentation has a description of this new curated data function's abilities. I have found it a bit slow, unfortunately, due to the extensive use of Entity[], but I guess this is the the way curated data functions are implemented these days.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.