Visualizing and analyzing a large bidirectional graph from adjacency matrix

Question

I have some data from simulations of neuronal synapses in the brain which I would like to analyze, and if possible visualize, using Mathematica. By analyzing I mean obtaining the clustering coefficient, average path length, degree distribution etc. I do not have much experience dealing with graph analysis in general, so your suggestions would be very helpful.

The number of the nodes is relatively large ($10^4 - 10^5$) but the edges are rather sparse. The edges are weighted, directed, and in addition, the weights are not symmetric (i.e. $w_{ij} \neq w_{ji}$ where $w_{ij}$ represents the weight of the edge between nodes $i$ and $j$).

The first question is what would be the best way to input these data. Particularly, I have two methods in mind: method one where I input the adjacency matrix which includes the weights. For example, in the case of $3$ nodes it would be something like

adjmatrix = { {0 , .7 , .3} , {.2 , 0 , .8} , {.6 , .4 , 0 } }

However, in my case this is (at least) a $10^4\times10^4$ matrix with many $0$ entries.

Method two, which I guess might be more efficient, is to store the adjacency list and the corresponding adjacency weights in separate files. For instance, in the above case, we can index the nodes as $a$, $b$, $c$ and have

adjlist = { { b , c } , { a , c} , {a , b} }

and

weights = { {.7,.3} , {.2,.8} , {.6,.4}}

When the number of nodes increases, it seems to me method two could be more efficient, but I am not sure how I would be able to use it in Mathematica.

The second question is about how to analyze the data (especially to find the clustering coefficient, mean path length, and outgoing degree distribution of the nodes) through each of the input methods above.

My third question is how to efficiently visualize these data. I have seen a few posts here regarding graph visualizations (for example this, this, this, and this) but I do not think these are particularly suitable for really large graphs. What I would prefer to demonstrate in the graph are the weights of the links and the degree of each node, maybe using colors etc.

I also understand this might not be the best representation method for the large number of nodes. I do not particularly need to show individual nodes/edges, but I was wondering maybe a coarse-grained representation exists that somehow communicates the overall structure.

PS
These are some sample data for a graph with $100$ nodes (in general, saving the data in the second format is much simpler, but generating the first format is also doable if that is more suitable):
method 1: adjacency matrix
method 2: adjacency list and the corresponding weights
(note that nodes are indexed from $0$ to $99$ and the first line of the adjacency list includes those nodes that node $0$ has a link to, the second line those that nodes $1$ has a link to and so on. The weight of each link is included in the weights file)

Answer 1

For the moment I can (somewhat) answer the first two questions.

Assuming you have the weights and adjacencies in separate lists like you showed in your question, you can construct the graph as follows:

adjlist = {{2, 3}, {1, 2}, {1, 2}};
weights = {{.7, .3}, {.2, .8}, {.6, .4}};

adjMat = SparseArray[Flatten@MapThread[{#1 -> #2[[1]], Reverse@#1 -> #2[[2]]} &, 
    {adjlist, weights}], {3,3}, Infinity];

graph = WeightedAdjacencyGraph[adjMat, EdgeLabels -> "EdgeWeight",
    VertexLabels -> {1->"a",2->"b",3->"c"}]

Mathematica interprets zeroes in the adjacency matrix as edges with zero weight, so you have to set the weights of absent edges to $\infty$ (there is more info on this here).

You can then use the built-in Mathematica functions for analysis:

Through[{MeanGraphDistance,GlobalClusteringCoefficient,
         VertexDegree,VertexOutDegree}[graph]]

{0.45, 1, {4, 4, 4}, {2, 2, 2}}

This also works well on larger examples:

n = 1000;

largeGraphAdjList = Join @@ Table[{{i, RandomInteger[{1, n}]}, {i, RandomInteger[{1, n}]}}, {i, 1, n}];
largeGraphWeights = Table[RandomReal[{0, 1}, {2}], {i, 1, 2n}];

largeGraphAdjMat = SparseArray[Flatten@MapThread[{#1 -> #2[[1]], Reverse@#1 -> #2[[2]]} &, 
     {largeGraphAdjList,largeGraphWeights}], {n, n}, Infinity];

largeGraph = WeightedAdjacencyGraph[largeGraphAdjMat];

Through[{MeanGraphDistance, MeanClusteringCoefficient, 
         Histogram@*VertexDegree, Histogram@*VertexOutDegree}[largeGraph]]

CommunityGraphPlot[largeGraph, EdgeStyle -> Directive[Arrowheads[0.01], AbsoluteThickness[0.1]]]

Regarding visualization, the answers in this link you posted seem to have some good ideas. For a coarse-grained representation, perhaps the CommunityGraphPlot style shown above is useful? I suppose it really depends on what feature of the graph you want to highlight.