# Fetching data from HTML source

I want to generate a couple of plots/graphs with Area51 statistics. Since Area51 doesn't work with the SE API, I'm forced to find another way to get the information I want.

That other way is with RegularExpression[] (or Mathematica's string patterns). Let's start with a specific example, then move on to a more general case.

Mathematica lets me fetch the source of an HTML page with

src=Import["http://area51.stackexchange.com/proposals/4470/martial-arts","Source"];


src now contains a string, which is the source code of the page.

What am I looking for?

The information that interests me is the value inside each of the blue squares, as well as its textual counterpart, namely Needs Work, Okay, or Excellent.

I would use something like

StringCases[a,
RegularExpression[
"regex"] ->
"$1"]  to get the data I want, but unfortunately I don't know how to write regexes. Side question: should I use Mathematica's string patterns, or RegularExpression[]? The more general case Once I can fetch the data above for one site, it's a piece of cake to get it for every proposal in beta. My goal is to be able to plot this data, to see how our site compares against the others concerning SE's top 5 health indicators. - ## 5 Answers I recommend that you import as an XMLObject, which represents structured XML data in a Mathematica-based format. info = Import[ "http://area51.stackexchange.com/proposals/4470/martial-arts", "XMLObject"];  You can access the parts of xml using Mathematica patterns, like so: labels = Cases[info, XMLElement[ "div", {"class" -> "site-health-label"}, label_] :> First[label], Infinity]; values = Cases[info, XMLElement[ "div", {"class" -> "site-health-value"}, value_] :> First[value], Infinity]; Grid[{labels, values}, Dividers -> All]  - +1, a straightforward way of accomplishing this. – rcollyer Mar 23 '12 at 2:59 +1, I have used this approach to get the list of followers/committers before the site went into beta. If anyone wants the code, my email is in my profile ;-) – Szabolcs Mar 23 '12 at 7:25 This might work for this case, but don't expect it to always work. HTML can't often be parsed using an XML parser because it's not XML compliant. HTML breaks a lot of rules, so don't be surprised if this stops working one day. – Travis Powell Mar 23 '12 at 17:40 Do you know of any way to force 404 pages to work for all sites? For example try the following Import["http://en.wikipedia.org/wiki/Oaeu", "XMLElement"] I am on W7 and M8 currently (being familiar with Linux I might use something like wget or curl as a quick workaround) but I figured there might be a built in way that is cross platform for Mathematica. I see M9 has URLFetch but that isn't exactly useful in M8 ;) – William Sep 13 '13 at 2:35 No one tried to solve this importing as plain text. So this is my answer using using only string patterns: src = Import["http://area51.stackexchange.com/proposals/4470/martial-arts", "Plaintext"]; With[{cmt = Whitespace~~comment:("Excellent"|"Needs Work"|"Okay")}, StringCases[ src, { grade:NumberString~~Whitespace~~"questions"~~Whitespace~~"per day"~~cmt -> {grade,comment}, percent:NumberString~~"%"~~Whitespace~~"answered"~~cmt -> {percent,comment}, avidUsers:NumberString~~Whitespace~~"avid users"~~Whitespace~~totalUsers:NumberString~~Whitespace~~"total users"~~cmt -> {avidUsers,totalUsers,comment}, answerRatio:NumberString~~Whitespace~~"answer ratio" ~~cmt -> {answerRatio,comment}, visitsPerDay:NumberString~~Whitespace~~"visits/day" ~~cmt -> {visitsPerDay,comment} } ] ]  And the output is: {{1.2,Needs Work},{98,Excellent},{37,349,Needs Work},{3.5,Excellent},{83,Needs Work}}  - After much fiddling... This function gathers the "vital signs" of every proposal, and returns everything in a list. proposalURL = "http://area51.stackexchange.com/?tab=beta&page="; getProposalData[main_] := Module[{b, srclist, pnumber, imp1, url, imp2, proplist, proplinks, fullproplinks, propxml, health, propname, fulldata}, srclist = {}; pnumber = 1; imp1[x_] := Import[srclist[[x]], "Source"]; b = 1; (*How many proposal pages?*) AppendTo[srclist, main <> ToString[b] <> "&pagesize=50"]; For[b = 1, Length[StringCases[imp1[pnumber], RegularExpression["No\\s+proposals\\s+in\\s+beta"]]] == 0, pnumber++, AppendTo[srclist, main <> ToString[pnumber] <> "&pagesize=50"]]; (*Gather proposal location from XML data*) url = DeleteDuplicates[srclist]; imp2[x_] := Import[url[[x]], "XMLObject"]; proplist = Flatten[Table[ Cases[imp2[i], XMLElement["div", {"class" -> "a51-summary"}, ___], Infinity], {i, 1, Length[url]}], 1]; proplinks = Table[proplist[[i]][[3]][[2]][[3]][[1]][[2]][[3]][[2]], {i, 1, Length[proplist]}]; (*Generate well-formed links and collect numerical values*) fullproplinks[x_] := Import["http://area51.stackexchange.com" <> proplinks[[x]], "XMLObject"]; propxml = Flatten[Table[ Cases[fullproplinks[i], XMLElement["div", {"class" -> "site-health-detail"}, ___], Infinity], {i, 1, Length[proplinks]}], 1]; health = ToExpression[ StringReplace[ Flatten[Table[ propxml[[i]][[3]][[2]][[3]][[2]][[3]], {i, 1, 5 Length[proplinks]}], 1], {"%" -> "", "," -> ""}]]; (*Gather proposal names*) propname = Flatten[Table[ StringCases[proplinks[[i]], RegularExpression["[^/]+$"]], {i, 1,

fulldata =
Partition[
Flatten[Riffle[health, propname, {1, 6 Length[proplinks], 6}]],
6]
]


To go from beta proposals to launched proposals, one needs only to change the proposalURL to http://area51.stackexchange.com/?tab=launched&page= and the first regex to No launched proposals.

The function is a hybdrid of the methods offered to me in the other answers. I don't quite know how I could have done it (as concisely) other than using both XML and regex. Thank you!

Charts

I've made some charts, but they're not very elegant. Have fun making the data talk.

As of March 24, 2012:

LAUNCHED

Questions per day

Avid users

Visits per day

BETA

Questions per day

Avid users

Visits per day

-
I like this. +1 Consider making the percent-answered graph span only the exhibited range. –  Mr.Wizard Mar 24 '12 at 21:12

Alright, follows are the regular expressions required to solve this. For further information on how these things work I would check out

Scroll down to the end of this question for an explanation on the various parts of this regular expression.

## Code

siteHealthRegexString[str_] :=
"(?s)div class=\\\"site-health-value\\\">([.,0-9%]+)</div>\\s+<div \
class=\\\"site-health-label\\\">" <> str <>
".*?<span class=\\\"site-health-status-([a-z]+)\\\"";

getSiteHealthValue[str_, siteText_] :=
StringCases[siteText,
RegularExpression[siteHealthRegexString[str]] -> {"$1", "$2"}]

getSiteHealth[url_String] :=
With[{site = Import[url, "Source"]},
"avid users", "total users", "answer ratio", "visits"}]]


Then look up the statistics for a given site by calling

getSiteHealth[
"http://area51.stackexchange.com/proposals/4470/martial-arts"]


Which results in:

  {{"1.4", "worrying"}, {"97%", "excellent"}, {"36", "worrying"},
{"344", "worrying"}, {"3.5", "excellent"}, {"82", "worrying"}}


If you would like to have the numerical results separate from the english results, just do a transpose like:

Transpose@getSiteHealth[
"http://area51.stackexchange.com/proposals/4470/martial-arts"]


Which results in:

 {{"1.4", "97%", "36", "344", "3.5", "82"},
{"worrying", "excellent", "worrying", "worrying", "excellent", "worrying"}}


## Regex Explanation

Basically the interesting elements of my regex up there are:

1. \\\" - used to escape a single double quote character.
2. [.0-9%] - a regex grouping matching the characters ., ',', 0, 1, ..., 9, %
3. (...) - indicates that the pattern inside the parens are to be returned as a "match". The first set of parens go to "$1", the second will go to "$2" and so forth.
4. \\s - matches any whitespace character. Including newlines, spaces and soforth.
5. + - indicates to match 1 or more of the proceeding element. You should note my use of this instruction twice in the regex. The first was to indicate that I wished to match more than one number/percent symbol/dot at a time. The second use is to indicate that I wish to match one or more whitespaces.
6. . is a wildcard indicating any character.
7. * instructs the parser to look for 0 or more of whatever pattern or group came immediately before it.
8. ? Indicates to the regular expression engine that the pattern should be matched as soon as it is found. (Otherwise called the "greedy" operator.) Without this, the regular expression engine will continue until the very last match "eating" up everything in between.
9. .*? thus means to look for zero or more instances of any character, including newlines, and stop matching as soon as it finds a suitable match for the remainder of the pattern. Without this all the entries would read "worrying" as they would all match to the very last item (corresponding to "visits/day") in the given site box.
10. (?s) is used to indicate that the . wildcard is to include newlines.

## Side Question

I'm not directly familiar with mathematica symbolic regular expressions. So I went ahead and asked in the MMA chat about the side question and was told, by @acl, that the implementation details listing state that the two are equivalent to the kernel. As such I vastly prefer the perl compatible regular expressions over the symbolic regular expressions as nearly every single programming language in existence understands them. With that aside, there may be compelling reasons to use the symbolic form, I'm just unaware of them.

-
about the side question: my personal theory is that regular expressions are actually better than MMA string patterns, but historically there have never been regular expressions in MMA before version 5.1. When they were finally introduced, some people were very happy (emacs user like me for example), but others had gotten used to the MMA string matching syntax and probably saw little reason to switch. –  Jens Mar 23 '12 at 4:13
Cool to see a fellow emacs user! That aside, the tidbit of history there adds a bit of context and makes sense. My other idea was some form of "enhanced" symbolic manipulations and such as done by something like cl-ppcre weitz.de/cl-ppcre . –  nixeagle Mar 23 '12 at 4:19
Gasp! No vim? / begins downvoting all of Jens' and nixeagle's posts... –  The Toad Mar 23 '12 at 4:26
It works for this particular case, but regexps will be a pain to maintain over time as the source changes. Plus, it can never be a completely generic solution, because HTML cannot be parsed with regular expressions –  F'x Mar 24 '12 at 23:00

To import as XMLObject is probably the cleanest approach, but since you asked for pattern matching I'll also post my answer which is similar to @nixeagle's but extracts instead the values together with their labels:

src=Import["http://area51.stackexchange.com/proposals/4470/martial-arts","Source"];
StringCases[src,
RegularExpression[
"<div\\s*class=\"site-health-value\">(.*)</div>\\s*<div\\s*class=\"site-health-label\">(.*)</div>"] -> {"$1", "$2"}]


With the nice explanation given by @nixeagle this could serve as another practice example.

The result looks something like this:

{{"1.4", "questions<br />per day"}, {"97%", "answered"}, {"36",
"avid users"}, {"344", "total users"}, {"3.5",