Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Is there a Mathematica API for the functions.wolfram site?
If there's not, has anyone implemented a web scraper for it?

For example it would be nice to be able to access http://functions.wolfram.com/07.23.17.0081.01 from within Mathematica by using something like

In[1]:= FunctionsWolfram["07.23.17.0081.01", InputForm]
Out[1]= Hypergeometric2F1[a,b,a+b-1/2,z]
        == Hypergeometric2F1[2 a-1,2 b-1,a+b-1/2,1/2 (1-Sqrt[1-z])]/Sqrt[1-z]

In[2]:= FunctionsWolfram["07.23.17.0081.01", RuleForm]
Out[2]= HoldPattern[Hypergeometric2F1[a_,b_,a_+b_-1/2,z_]] 
        :> Hypergeometric2F1[2 a-1,2 b-1,a+b-1/2,1/2 (1-Sqrt[1-z])]/Sqrt[1-z]

In[3]:= FunctionsWolfram["07.23.17.0081.01", TraditionalForm]
Out[3]=

Image_from_http://functions.wolfram.com/07.23.17.0081.01

share|improve this question
add comment

migrated from stackoverflow.com Jan 29 '12 at 3:36

This question came from our site for professional and enthusiast programmers.

3 Answers

The site is not terribly conducive to scraping as the HTML is "noisy" and looks like WRI might change the format at the drop of a hat. Throwing caution to the wind...

scrapeWolframFunction[id_] :=
  Import["http://functions.wolfram.com/" ~~ id, "XMLObject"] //
  Cases[
    #
  , XMLElement["p", {___, "class" -> "CitationInfo", ___}, body_] :> body
  , Infinity
  ] & //
  ToExpression[#[[1, 1]], InputForm, HoldForm] &

The function assumes that the first CitationInfo paragraph contains the InputForm. This assumption appears to hold true for the moment.

Sample use:

In[24]:= scrapeWolframFunction["07.23.17.0081.01"]
Out[24]= Hypergeometric2F1[a,b,a+b-1/2,z]==Hypergeometric2F1[2 a-1,2 b-1,a+b-1/2,1/2 (1-Sqrt[1-z])]/Sqrt[1-z]

In[25]:= scrapeWolframFunction["01.06.02.0001.01"]
Out[25]= Sin[z]==(E^(I z)-E^(-I z))/(2 I)

In[26]:= scrapeWolframFunction["06.25.27.0004.01"]
Out[26]= Erf[z]==(1+I) (FresnelC[((1-I) z)/Sqrt[\[Pi]]]-I FresnelS[((1-I) z)/Sqrt[\[Pi]]])
share|improve this answer
3  
Mathematica's built-in HTML parser is TagSoup which is very good at handling HTML as it is found in the wild. It is worth the effort to persevere through XMLObject if one has any need to parse either HTML or XML. –  WReach Jun 2 '11 at 3:42
add comment

Here is a shameless plug for my HTML parser posted here. The code is a bit long to reproduce here, the only change to it I'd do is to replace the function processPosList with this code:

processPosList::unmatched = "Unmatched lists `1` enountered!";
processPosList[{openlist_List, closelist_List}] := 
  Module[{opengroup, closegroup, poslist}, 
  {opengroup, closegroup} = groupPositions /@ {openlist, closelist};
   poslist = Transpose[Transpose[Sort[#]] & /@ {opengroup, closegroup}];
   If[UnsameQ @@ poslist[[1]], Return[(Message[
       processPosList::unmatched , {openlist, closelist}]; {})], 
   poslist = Transpose[{poslist[[1, 1]], Transpose /@ Transpose[poslist[[2]]]}]]];

which will issue a message when some parts can not be parsed instead of printing the details (as the original code does). I must warn that my parser for some reason can not fully parse the Wolfram Functions pages (either they are ill-formed or my parser contains bugs), but it will parse enough for our purposes. Here is a simple web-scraper based on it and on a few observations about the typical format of the page:

Clear[getForms];
getForms[url_String] := 
 Quiet@ Cases[postProcess@parseText[Import[url, "Text"]],
     pContainer[attribContainer[" class='CitationInfo'"], x__String] :> 
        StringJoin@x, Infinity] //. 
       x_String :>  StringReplace[ x, {""" | "quot;" :> "\"", "&" :> "", 
             "&lt;" | "&lt" :> "<", "&gt;" | "&gt" :> ">", "\n" :> " "}];


Clear[formsOk, getInputForm, getStandardForm, getRuleForm];
formsOk[forms_] := Length[forms] == 5;
getInputForm[forms_?formsOk] := ToExpression[forms[[1]], InputForm];
getStandardForm[forms_?formsOk] := ToExpression[First@ToExpression[forms[[2]]], StandardForm];
getRuleForm[forms_?formsOk] := ToExpression[First@ToExpression[forms[[4]]]];
getInputForm[__] = getStandardForm[__] = getRuleForm[__] = $Failed;

I can not say how fragile this is, probably rather fragile. Here is an example of use:

In[277]:= 
forms = getForms["http://functions.wolfram.com/07.23.17.0084.01"];
Through[{getInputForm,getStandardForm,getRuleForm}[forms]]

Out[278]= {Hypergeometric2F1[a,b,-(1/2)+a+b,z]==((Sqrt[1-z]-Sqrt[-z])^(1-2 a) 
  Hypergeometric2F1[-1+2 a,-1+a+b,-2+2 a+2 b,2 z+2 Sqrt[-z+z^2]])/Sqrt[1-z]/;Re[z]>1/2,
  Hypergeometric2F1[a,b,-(1/2)+a+b,z]==((Sqrt[1-z]-Sqrt[-z])^(1-2 a) 
  Hypergeometric2F1[-1+2 a,-1+a+b,-2+2 a+2 b,2 z+2 Sqrt[-z+z^2]])/Sqrt[1-z]/;Re[z]>1/2,
  HoldPattern[Hypergeometric2F1[a_,b_,a_+b_-1/2,z_]]:>((Sqrt[1-z]-Sqrt[-z])^(1-2 a) 
  Hypergeometric2F1[2 a-1,a+b-1,2 a+2 b-2,2 Sqrt[z^2-z]+2 z])/Sqrt[1-z]/;Re[z]/2}

I tested on about 10 different formulas, and this worked fine, but of course this is not an extensive test, so most likely this will not always work.

share|improve this answer
    
Thanks Leonid! I'll have a closer look at this tomorrow. I wouldn't be too concerned about its fragility - it's probably a lot more robust than my solution... –  Simon May 31 '11 at 11:57
    
@Simon For your solution, the heavy lifting is done with Import, which does a pretty good job of importing in default format (HTML here). My solution parses HTML and looks at certain pattern in the parsed document. For the case at hand, your solution seems much more robust and adequate, so I am tempted to delete mine - perhaps will give it a day or two... –  Leonid Shifrin May 31 '11 at 12:08
add comment

Here's a quick scraper that I built (after asking the question). So far it has minimal error checking. Also note that InputForm, StandardForm and MathMLForm should all yield the same expressions.

FunctionsWolframIDQ[id_String]:=StringMatchQ[id,
  RegularExpression["\\d\\d\\.\\d\\d\\.\\d\\d\\.\\d\\d\\d\\d\\.\\d\\d"]]
FWIDQ=FunctionsWolframIDQ;

FunctionsWolfram[id_String?FWIDQ]:=FunctionsWolfram[id,InputForm]

FunctionsWolfram[id_String?FWIDQ,All] := FunctionsWolfram[id,All] = 
  Module[{imp=Import["http://functions.wolfram.com/"<>id]}, StringSplit[imp,
    "Input Form"|"Standard Form"|"MathML Form"|"Rule Form"|"Date Added"]]

FunctionsWolfram[id_String?FWIDQ,InputForm] := 
  ToExpression@StringTrim@FunctionsWolfram[id,All][[2]]

FunctionsWolfram[id_String?FWIDQ,StandardForm] := 
  ToExpression@First@ToExpression@StringTrim@FunctionsWolfram[id,All][[3]]

FunctionsWolfram[id_String?FWIDQ,MathMLForm] :=
  ToExpression[StringTrim@FunctionsWolfram[id,All][[4]],MathMLForm]

FunctionsWolfram[id_String?FWIDQ,RuleForm] := 
  ToExpression@First@ToExpression@StringTrim@FunctionsWolfram[id,All][[5]]

FunctionsWolfram[id_String?FWIDQ, TraditionalForm] := 
  TraditionalForm[FunctionsWolfram[id, InputForm]]

It works as advertised in the question.

share|improve this answer
    
This was just a really quick solution that just reads a functions site in as a long string and then splits it up and spits out the appropriate parts. I left the answer "deleted" until I was ready to call it a night and not check SO until morning! –  Simon May 31 '11 at 11:56
    
Nice. I was already doing messy pattern matching in the html code read in using Import[...,"Text"]: StringReplace[text, ___ ~~ "<font class='CitationLabel' size='-1'>Input Form</font>" ~~ Shortest[_] ~~ "<p class='CitationInfo'>" ~~ Shortest[inputForm ~~ "</p>"] ~~ ___ :> inputForm]. This is much cleaner –  Sjoerd C. de Vries May 31 '11 at 12:25
1  
@Sjoerd Re your comment. Have you read this excellent post: stackoverflow.com/questions/1732348/…? –  belisarius Jun 3 '11 at 3:47
    
It seems to me this great function would gain a lot if it could take WildCards, such as FunctionWolfram["HyperGeometric2F1"] so that it would returns all identities related to HyperGeometric2F1 funtions. Once the index of such entries identified one could imagine using something like Table[FunctionsWolfram[ "07.23.17." <> IntegerString[i, 10, 4] <> ".01"], {i, 4}]? The purpose of this would be to produce transformation rules that the user could map onto given mathematica results to suit his/her need. –  chris Apr 28 '12 at 18:54
    
@chris: That does sound great! The functions site used to have and advanced search form that used standard Mathematica patterns... but I can't find it on the site anymore. Feel free to develop something more useful than what has been posted here and I'll accept your answer (as I couldn't choose between the three answers already posted). –  Simon Apr 29 '12 at 4:12
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.