Fetching data from HTML source

Question

I want to generate a couple of plots/graphs with Area 51 statistics. Since Area 51 doesn't work with the SE API, I'm forced to find another way to get the information I want.

That other way is with RegularExpression[] (or Mathematica's string patterns). Let's start with a specific example, then move on to a more general case.

Mathematica lets me fetch the source of an HTML page with

src=Import["http://area51.stackexchange.com/proposals/4470/martial-arts","Source"];

src now contains a string, which is the source code of the page.

What am I looking for?

The information that interests me is the value inside each of the blue squares, as well as its textual counterpart, namely Needs Work, Okay, or Excellent.

I would use something like

StringCases[a, 
 RegularExpression[
   "regex"] ->
   "$1"]

to get the data I want, but unfortunately I don't know how to write regexes.

Side question: should I use Mathematica's string patterns, or RegularExpression[]?

The more general case

Once I can fetch the data above for one site, it's a piece of cake to get it for every proposal in beta. My goal is to be able to plot this data, to see how our site compares against the others concerning SE's top 5 health indicators.

Answer 1

I recommend that you import as an XMLObject, which represents structured XML data in a Mathematica-based format.

info = Import[
   "http://area51.stackexchange.com/proposals/4470/martial-arts", 
   "XMLObject"];

You can access the parts of xml using Mathematica patterns, like so:

labels = Cases[info, XMLElement[
  "div", {"class" -> "site-health-label"}, label_] :> 
  First[label], Infinity];
values = Cases[info, XMLElement[
  "div", {"class" -> "site-health-value"}, value_] :> 
   First[value], Infinity];
Grid[{labels, values}, Dividers -> All]

Answer 2

Alright, follows are the regular expressions required to solve this. For further information on how these things work I would check out

• the regular expression documentation on the wolfram docs site.
• As far as how regular expressions themselves work, check out the perl regular expression man page. This is where I learned how to write regexes back when I was in 8th grade or so ;).

Scroll down to the end of this question for an explanation on the various parts of this regular expression.

Code

siteHealthRegexString[str_] := 
  "(?s)div class=\\\"site-health-value\\\">([.,0-9%]+)</div>\\s+<div \
class=\\\"site-health-label\\\">" <> str <> 
   ".*?<span class=\\\"site-health-status-([a-z]+)\\\"";

getSiteHealthValue[str_, siteText_] :=
 StringCases[siteText,
  RegularExpression[siteHealthRegexString[str]] -> {"$1", "$2"}]

getSiteHealth[url_String] :=
 With[{site = Import[url, "Source"]},
  Map[(getSiteHealthValue[#, site][[1]] &), {"questions", "answered", 
    "avid users", "total users", "answer ratio", "visits"}]]

Then look up the statistics for a given site by calling

getSiteHealth[
  "http://area51.stackexchange.com/proposals/4470/martial-arts"]

Which results in:

  {{"1.4", "worrying"}, {"97%", "excellent"}, {"36", "worrying"}, 
   {"344", "worrying"}, {"3.5", "excellent"}, {"82", "worrying"}}

If you would like to have the numerical results separate from the english results, just do a transpose like:

Transpose@getSiteHealth[
 "http://area51.stackexchange.com/proposals/4470/martial-arts"]

Which results in:

 {{"1.4", "97%", "36", "344", "3.5", "82"}, 
  {"worrying", "excellent", "worrying", "worrying", "excellent", "worrying"}}

Regex Explanation

Basically the interesting elements of my regex up there are:

1. \\\" - used to escape a single double quote character.
2. [.0-9%] - a regex grouping matching the characters ., ',', 0, 1, ..., 9, %
3. (...) - indicates that the pattern inside the parens are to be returned as a "match". The first set of parens go to "$1", the second will go to "$2" and so forth.
4. \\s - matches any whitespace character. Including newlines, spaces and soforth.
5. + - indicates to match 1 or more of the proceeding element. You should note my use of this instruction twice in the regex. The first was to indicate that I wished to match more than one number/percent symbol/dot at a time. The second use is to indicate that I wish to match one or more whitespaces.
6. . is a wildcard indicating any character.
7. * instructs the parser to look for 0 or more of whatever pattern or group came immediately before it.
8. ? Indicates to the regular expression engine that the pattern should be matched as soon as it is found. (Otherwise called the "greedy" operator.) Without this, the regular expression engine will continue until the very last match "eating" up everything in between.
9. .*? thus means to look for zero or more instances of any character, including newlines, and stop matching as soon as it finds a suitable match for the remainder of the pattern. Without this all the entries would read "worrying" as they would all match to the very last item (corresponding to "visits/day") in the given site box.
10. (?s) is used to indicate that the . wildcard is to include newlines.

Side Question

I'm not directly familiar with mathematica symbolic regular expressions. So I went ahead and asked in the MMA chat about the side question and was told, by @acl, that the implementation details listing state that the two are equivalent to the kernel. As such I vastly prefer the perl compatible regular expressions over the symbolic regular expressions as nearly every single programming language in existence understands them. With that aside, there may be compelling reasons to use the symbolic form, I'm just unaware of them.

Answer 3

After much fiddling... This function gathers the "vital signs" of every proposal, and returns everything in a list.

proposalURL = "http://area51.stackexchange.com/?tab=beta&page=";

getProposalData[main_] := 
 Module[{b, srclist, pnumber, imp1, url, imp2, proplist, proplinks, 
   fullproplinks, propxml, health, propname, fulldata},
  srclist = {};
  pnumber = 1;
  imp1[x_] := Import[srclist[[x]], "Source"];
  b = 1;

  (*How many proposal pages?*)

  AppendTo[srclist, main <> ToString[b] <> "&pagesize=50"];
  For[b = 1, 
   Length[StringCases[imp1[pnumber], 
      RegularExpression["No\\s+proposals\\s+in\\s+beta"]]] == 0, 
   pnumber++, 
   AppendTo[srclist, main <> ToString[pnumber] <> "&pagesize=50"]];

  (*Gather proposal location from XML data*)

  url = DeleteDuplicates[srclist];
  imp2[x_] := Import[url[[x]], "XMLObject"];

  proplist = 
   Flatten[Table[
     Cases[imp2[i], 
      XMLElement["div", {"class" -> "a51-summary"}, ___], 
      Infinity], {i, 1, Length[url]}], 1];

  proplinks = 
   Table[proplist[[i]][[3]][[2]][[3]][[1]][[2]][[3]][[2]], {i, 1, 
     Length[proplist]}];

  (*Generate well-formed links and collect numerical values*)

  fullproplinks[x_] := 
   Import["http://area51.stackexchange.com" <> proplinks[[x]], 
    "XMLObject"];
  propxml = 
   Flatten[Table[
     Cases[fullproplinks[i], 
      XMLElement["div", {"class" -> "site-health-detail"}, ___], 
      Infinity], {i, 1, Length[proplinks]}], 1];
  health = 
   ToExpression[
    StringReplace[
     Flatten[Table[
       propxml[[i]][[3]][[2]][[3]][[2]][[3]], {i, 1, 
        5 Length[proplinks]}], 1], {"%" -> "", "," -> ""}]];

  (*Gather proposal names*)

  propname = 
   Flatten[Table[
     StringCases[proplinks[[i]], RegularExpression["[^/]+$"]], {i, 1, 
      Length[proplinks]}]];

  (*Riffle lists, tadda!*)

  fulldata = 
   Partition[
    Flatten[Riffle[health, propname, {1, 6 Length[proplinks], 6}]], 
    6]
  ]

To go from beta proposals to launched proposals, one needs only to change the proposalURL to http://area51.stackexchange.com/?tab=launched&page= and the first regex to No launched proposals.

The function is a hybdrid of the methods offered to me in the other answers. I don't quite know how I could have done it (as concisely) other than using both XML and regex. Thank you!

Charts

I've made some charts, but they're not very elegant. Have fun making the data talk.

As of March 24, 2012:

---

LAUNCHED

Questions per day

Percentage of answered questions

Avid users

Answer ratio

Visits per day

---

BETA

Questions per day

Percentage of answered questions

Avid users

Answer ratio

Visits per day

Answer 4

To import as XMLObject is probably the cleanest approach, but since you asked for pattern matching I'll also post my answer which is similar to @nixeagle's but extracts instead the values together with their labels:

src=Import["http://area51.stackexchange.com/proposals/4470/martial-arts","Source"];
StringCases[src, 
 RegularExpression[
   "<div\\s*class=\"site-health-value\">(.*)</div>\\s*<div\\s*class=\"site-health-label\">(.*)</div>"] -> {"$1", "$2"}]

With the nice explanation given by @nixeagle this could serve as another practice example.

The result looks something like this:

{{"1.4", "questions<br />per day"}, {"97%", "answered"}, {"36", 
  "avid users"}, {"344", "total users"}, {"3.5", 
  "answer ratio"}, {"82", "visits/day"}}

Answer 5

No one tried to solve this importing as plain text. So this is my answer using using only string patterns:

src = Import["http://area51.stackexchange.com/proposals/4470/martial-arts", 
             "Plaintext"];

With[{cmt = Whitespace~~comment:("Excellent"|"Needs Work"|"Okay")},
    StringCases[
        src,
        {
                grade:NumberString~~Whitespace~~"questions"~~Whitespace~~"per day"~~cmt -> {grade,comment},
              percent:NumberString~~"%"~~Whitespace~~"answered"~~cmt -> {percent,comment},
            avidUsers:NumberString~~Whitespace~~"avid users"~~Whitespace~~totalUsers:NumberString~~Whitespace~~"total users"~~cmt -> {avidUsers,totalUsers,comment},
          answerRatio:NumberString~~Whitespace~~"answer ratio" ~~cmt -> {answerRatio,comment},
         visitsPerDay:NumberString~~Whitespace~~"visits/day"   ~~cmt -> {visitsPerDay,comment}
        }
    ]
]

And the output is:

{{1.2,Needs Work},{98,Excellent},{37,349,Needs Work},{3.5,Excellent},{83,Needs Work}}