Recently Wolfram|Alpha announced that you could use it to analyze your Facebook site. How could I do a network analysis of questioners and answers here on Mathematica.SE?

  • $\begingroup$ This would be one starting point, although there's a newer version of the API which might affect the code a bit. $\endgroup$ Sep 20 '12 at 20:38
  • $\begingroup$ +1 I'd love to see some neat tools posted in this thread. $\endgroup$ Sep 20 '12 at 20:51
  • $\begingroup$ Although very interesting, I'm not sure whether it fits the site's Q&A format very well. $\endgroup$ Sep 20 '12 at 21:51
  • 5
    $\begingroup$ I think the question would be just fine if it merely focused on the process of generating that network and representing it as a Mathematica data structure (such as a directed graph, perhaps with suitable vertex and edge labels carrying attributes like time, votes, tags, and whatever). How to analyze it could occupy many more questions! $\endgroup$
    – whuber
    Sep 20 '12 at 23:51
  • 1
    $\begingroup$ Yes, the API works. See this question and this answer to a question on meta. I never got around to linking posts and people though - it would be interesting, but probably requires a Stack API key. $\endgroup$
    – Verbeia
    Sep 21 '12 at 0:39

I hope this thread can become a place for code that takes diverse angles on site analysis. Turns out our site connectivity looks like a supernova explosion, so I decided to share the pretty image. The original code can be found in A Speed Date with Mathematica by Jon McLoone on the liked page. Here are some connectivity data, accessed through a simple crawler. Let's create a webcrawler to find linkages between MSE URL and all of its hyperlinks:

webcrawler[rooturl_, depth_] := Flatten[Rest[NestList[Union[Flatten[Thread[# -> 
Import[#, "Hyperlinks"]] & /@     Last /@ #]] &, {"" -> rooturl}, depth]]];

MSEdata = Quiet@webcrawler["http://stackexchange.com/", 2];

this takes a few minutes to run. Style for the graph:

style = {VertexStyle -> White, VertexShapeFunction -> "Point", 
   EdgeStyle -> Directive[Opacity[.1], Hue[.15, .5, .8]], 
   Background -> Black, EdgeShapeFunction -> (Line[#1] &), 
   ImageSize -> 1500};

Visualize the result by generating a graph of networks:

Graph[MSEdata, style]

enter image description here

A quite beautiful image of MSE positioned among other SE sites. Meaning with the chosen depth of crawler 2 these data start from MSE as a root, but there is enough depth to spread into other sites. Some analysis now. Using the HITSCentrality function, find the HITS hub and authority centralities for the graph. Link the pages with their HITS centralities. Sort the pages according to their degree centralities. Here is the top 50 list:

pagesHITS = Transpose[{VertexList[g], HITSCentrality[g][[1]]}];
sortedpages = Sort[pagesHITS, #1[[2]] > #2[[2]] &];
TableForm[MapThread[Prepend, {sortedpages[[1 ;; 55]], Range[55]}]]

enter image description here enter image description here

So we can see our (growing ;-) ) significance at current moment - we already top some other sites. We could verify our place among other sites in another way using WolframAlpha function and going from the top SE domain - it gives about same result in terms of our rank:

SubDomData =  WolframAlpha["http://stackexchange.com/", 
 {{"WebSiteStatisticsPod:InternetData", 2}, "ComputableData"}, 
   PodStates -> {"WebSiteStatisticsPod:InternetData__Subdomains", 

Grid[MapThread[Prepend, {SubDomData, Range[Length[SubDomData]]}], 
 Frame -> All, Alignment -> Left]

enter image description here enter image description here

Note if we would use as a root the other site, like SE: we would get a different graph, so the graph above, even so it seeps into other domains, does reflect on how sites are linked from MSE point of view. This is SE as root http://stackexchange.com/

enter image description here

  • 1
    $\begingroup$ Hmm... I'd be a bit hesitant to follow down this path, because most of those linkages you see might not be genuine. For instance, every graduated site is linked to from every other graduated site's footer, so all of those will have to be filtered out somehow. $\endgroup$
    – rm -rf
    Sep 21 '12 at 20:13
  • $\begingroup$ @R.M You are right, high chance that the symmetry in the MSE graph could be due to the footer - other sites look like this too. I just wanted to put the idea out - if someone could post a better filtered code, or add an edit to mine - I'd be totally fine with it. $\endgroup$ Sep 21 '12 at 20:25
  • $\begingroup$ Writing a webcrawler can be a bit tricky. I've got a webpage (facstaff.unca.edu/mcmcclur/class/Seminar/Pagerank/nutch) that describes how to run a crawl using the open source webcrawler Nutch and then import the result. Might be able to deal with R.M.'s concern that way. $\endgroup$ Sep 21 '12 at 21:25
  • $\begingroup$ @MarkMcClure Thanks for pointing this out, Mark. I hope you or someone can get it working - would love to visualize the data in mma. $\endgroup$ Sep 21 '12 at 21:41
  • $\begingroup$ @VitaliyKaurov - This is kind of approach I was thinking of but didn't really know how to do. I hesitate to accept this wonderful answer only because it seems people want to continue the thread. Me too! Also, I'm interested in the connections within MMa SE. $\endgroup$ Sep 21 '12 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.