Recently Wolfram|Alpha announced that you could use it to analyze your Facebook site. How could I do a network analysis of questioners and answers here on Mathematica.SE?
$\begingroup$
$\endgroup$
9
-
$\begingroup$ This would be one starting point, although there's a newer version of the API which might affect the code a bit. $\endgroup$– Brett ChampionCommented Sep 20, 2012 at 20:38
-
$\begingroup$ +1 I'd love to see some neat tools posted in this thread. $\endgroup$– Vitaliy KaurovCommented Sep 20, 2012 at 20:51
-
$\begingroup$ Although very interesting, I'm not sure whether it fits the site's Q&A format very well. $\endgroup$– Dr. belisariusCommented Sep 20, 2012 at 21:51
-
5$\begingroup$ I think the question would be just fine if it merely focused on the process of generating that network and representing it as a Mathematica data structure (such as a directed graph, perhaps with suitable vertex and edge labels carrying attributes like time, votes, tags, and whatever). How to analyze it could occupy many more questions! $\endgroup$– whuberCommented Sep 20, 2012 at 23:51
-
1$\begingroup$ Yes, the API works. See this question and this answer to a question on meta. I never got around to linking posts and people though - it would be interesting, but probably requires a Stack API key. $\endgroup$– VerbeiaCommented Sep 21, 2012 at 0:39
|
Show 4 more comments
1 Answer
$\begingroup$
$\endgroup$
8
I hope this thread can become a place for code that takes diverse angles on site analysis. Turns out our site connectivity looks like a supernova explosion, so I decided to share the pretty image. The original code can be found in A Speed Date with Mathematica by Jon McLoone on the liked page. Here are some connectivity data, accessed through a simple crawler. Let's create a webcrawler to find linkages between MSE URL and all of its hyperlinks:
webcrawler[rooturl_, depth_] := Flatten[Rest[NestList[Union[Flatten[Thread[# ->
Import[#, "Hyperlinks"]] & /@ Last /@ #]] &, {"" -> rooturl}, depth]]];
MSEdata = Quiet@webcrawler["http://stackexchange.com/", 2];
this takes a few minutes to run. Style for the graph:
style = {VertexStyle -> White, VertexShapeFunction -> "Point",
EdgeStyle -> Directive[Opacity[.1], Hue[.15, .5, .8]],
Background -> Black, EdgeShapeFunction -> (Line[#1] &),
ImageSize -> 1500};
Visualize the result by generating a graph of networks:
Graph[MSEdata, style]
A quite beautiful image of MSE positioned among other SE sites. Meaning with the chosen depth of crawler 2 these data start from MSE as a root, but there is enough depth to spread into other sites. Some analysis now. Using the HITSCentrality function, find the HITS hub and authority centralities for the graph. Link the pages with their HITS centralities. Sort the pages according to their degree centralities. Here is the top 50 list:
pagesHITS = Transpose[{VertexList[g], HITSCentrality[g][[1]]}];
sortedpages = Sort[pagesHITS, #1[[2]] > #2[[2]] &];
TableForm[MapThread[Prepend, {sortedpages[[1 ;; 55]], Range[55]}]]
So we can see our (growing ;-) ) significance at current moment - we already top some other sites. We could verify our place among other sites in another way using WolframAlpha function and going from the top SE domain - it gives about same result in terms of our rank:
SubDomData = WolframAlpha["http://stackexchange.com/",
{{"WebSiteStatisticsPod:InternetData", 2}, "ComputableData"},
PodStates -> {"WebSiteStatisticsPod:InternetData__Subdomains",
"WebSiteStatisticsPod:InternetData__Subdomains_More"}];
Grid[MapThread[Prepend, {SubDomData, Range[Length[SubDomData]]}],
Frame -> All, Alignment -> Left]
Note if we would use as a root the other site, like SE: we would get a different graph, so the graph above, even so it seeps into other domains, does reflect on how sites are linked from MSE point of view. This is SE as root http://stackexchange.com/
answered Sep 21, 2012 at 20:04
-
1$\begingroup$ Hmm... I'd be a bit hesitant to follow down this path, because most of those linkages you see might not be genuine. For instance, every graduated site is linked to from every other graduated site's footer, so all of those will have to be filtered out somehow. $\endgroup$– rm -rf ♦Commented Sep 21, 2012 at 20:13
-
$\begingroup$ @R.M You are right, high chance that the symmetry in the MSE graph could be due to the footer - other sites look like this too. I just wanted to put the idea out - if someone could post a better filtered code, or add an edit to mine - I'd be totally fine with it. $\endgroup$ Commented Sep 21, 2012 at 20:25
-
$\begingroup$ Writing a webcrawler can be a bit tricky. I've got a webpage (facstaff.unca.edu/mcmcclur/class/Seminar/Pagerank/nutch) that describes how to run a crawl using the open source webcrawler Nutch and then import the result. Might be able to deal with R.M.'s concern that way. $\endgroup$ Commented Sep 21, 2012 at 21:25
-
$\begingroup$ @MarkMcClure Thanks for pointing this out, Mark. I hope you or someone can get it working - would love to visualize the data in mma. $\endgroup$ Commented Sep 21, 2012 at 21:41
-
$\begingroup$ @VitaliyKaurov - This is kind of approach I was thinking of but didn't really know how to do. I hesitate to accept this wonderful answer only because it seems people want to continue the thread. Me too! Also, I'm interested in the connections within MMa SE. $\endgroup$ Commented Sep 21, 2012 at 22:13