Use Mathematica to do a network analysis of Mathematica.SE

Question

Recently Wolfram|Alpha announced that you could use it to analyze your Facebook site. How could I do a network analysis of questioners and answers here on Mathematica.SE?

Answer 1

I hope this thread can become a place for code that takes diverse angles on site analysis. Turns out our site connectivity looks like a supernova explosion, so I decided to share the pretty image. The original code can be found in A Speed Date with Mathematica by Jon McLoone on the liked page. Here are some connectivity data, accessed through a simple crawler. Let's create a webcrawler to find linkages between MSE URL and all of its hyperlinks:

webcrawler[rooturl_, depth_] := Flatten[Rest[NestList[Union[Flatten[Thread[# -> 
Import[#, "Hyperlinks"]] & /@     Last /@ #]] &, {"" -> rooturl}, depth]]];

MSEdata = Quiet@webcrawler["http://stackexchange.com/", 2];

this takes a few minutes to run. Style for the graph:

style = {VertexStyle -> White, VertexShapeFunction -> "Point", 
   EdgeStyle -> Directive[Opacity[.1], Hue[.15, .5, .8]], 
   Background -> Black, EdgeShapeFunction -> (Line[#1] &), 
   ImageSize -> 1500};

Visualize the result by generating a graph of networks:

Graph[MSEdata, style]

A quite beautiful image of MSE positioned among other SE sites. Meaning with the chosen depth of crawler 2 these data start from MSE as a root, but there is enough depth to spread into other sites. Some analysis now. Using the HITSCentrality function, find the HITS hub and authority centralities for the graph. Link the pages with their HITS centralities. Sort the pages according to their degree centralities. Here is the top 50 list:

pagesHITS = Transpose[{VertexList[g], HITSCentrality[g][[1]]}];
sortedpages = Sort[pagesHITS, #1[[2]] > #2[[2]] &];
TableForm[MapThread[Prepend, {sortedpages[[1 ;; 55]], Range[55]}]]

So we can see our (growing ;-) ) significance at current moment - we already top some other sites. We could verify our place among other sites in another way using WolframAlpha function and going from the top SE domain - it gives about same result in terms of our rank:

SubDomData =  WolframAlpha["http://stackexchange.com/", 
 {{"WebSiteStatisticsPod:InternetData", 2}, "ComputableData"}, 
   PodStates -> {"WebSiteStatisticsPod:InternetData__Subdomains", 
     "WebSiteStatisticsPod:InternetData__Subdomains_More"}];

Grid[MapThread[Prepend, {SubDomData, Range[Length[SubDomData]]}], 
 Frame -> All, Alignment -> Left]

Note if we would use as a root the other site, like SE: we would get a different graph, so the graph above, even so it seeps into other domains, does reflect on how sites are linked from MSE point of view. This is SE as root http://stackexchange.com/