# Sorting Data by Differences between Columns

In Band Structure calculations one gets rather extensive data of Energy Values (Eigenvalues) and the associated Eigenvectors out of the solution of an Eigenequation of e.g. the form H.$\psi$=E.$\psi$. Solving this equation yield usually Eigenvalues/Eigenvector pairs sorted by decreasing Eigenvalues.

An example of such an dataset is shown here

data={{{-9.16206, -7.58803, -4.31229, -2.85908, 4.0047, 4.33906, 6.60042,
7.32482}, {{0.659095 - 0.0205512 I, -0.0074541 + 0.000232426 I,
0.0000771943 + 0.0608106 I, -0.00371154 - 0.0606973 I,
0.480582 + 0.451518 I,
0.0054352 + 0.00510649 I, -0.040295 + 0.0455439 I,
0.042945 - 0.0430542 I}, {-0.569797 + 0.159891 I, -0.132206 +
0.0370984 I, -0.146127 + 0.0289249 I, -0.139842 + 0.0513211 I,
0.515967 + 0.289847 I, -0.119716 - 0.0672513 I, -0.135173 -
0.062594 I, -0.12378 - 0.0828745 I}, {0.022029 +
0.102154 I, -0.129324 - 0.599706 I,
0.323283 - 0.408073 I, -0.462729 - 0.238574 I,
0.056657 - 0.0878108 I,
0.332611 - 0.515502 I, -0.158502 - 0.495896 I,
0.517147 - 0.0599557 I}, {0.00135768 - 0.0109025 I,
0.0677891 - 0.544363 I, -0.285998 + 0.275605 I,
0.209669 + 0.33733 I,
0.00866928 - 0.00674923 I, -0.432857 + 0.336988 I,
0.0902696 - 0.386787 I,
0.397113 + 0.00734894 I}, {0.478682 - 0.125004 I,
0.100588 - 0.0262677 I, -0.332849 + 0.337306 I, -0.455272 -
0.131494 I, -0.426871 - 0.250088 I,
0.0897003 + 0.0525522 I, -0.228946 - 0.414906 I, -0.473871 +
0.0031511 I}, {-0.0926624 - 0.0934746 I,
0.313489 + 0.316237 I, -0.282601 + 0.246448 I,
0.248905 - 0.28044 I, 0.000574334 - 0.131619 I,
0.00194305 - 0.445284 I, -0.374303 + 0.0222986 I,
0.374094 + 0.0255643 I}, {-0.448984 + 0.190802 I, -0.354137 +
0.150495 I, -0.282038 + 0.29955 I, -0.411402 -
0.00486289 I, -0.452397 - 0.182562 I, 0.356829 + 0.143996 I,
0.287467 + 0.294344 I,
0.411245 - 0.0123828 I}, {0.0966639 + 0.257887 I, -0.242141 -
0.646001 I, -0.210795 + 0.17766 I, 0.275637 - 0.0046696 I,
0.114002 - 0.250705 I, 0.285572 - 0.628011 I,
0.198206 + 0.191603 I, -0.274679 -
0.0234301 I}}}, {{-9.29415, -7.50801, -4.24412, -2.68539,
3.85009, 4.18365, 6.82186,
7.26021}, {{-0.668047 + 0.0375069 I,
0.00216553 - 0.000121582 I, -0.00859939 -
0.0594807 I, -0.00188735 + 0.0600695 I, -0.498902 -
0.445859 I, -0.00161723 - 0.00144529 I,
0.0438101 - 0.041141 I, -0.0359785 + 0.0481399 I}, {-0.548008 +
0.196971 I, -0.158467 + 0.0569581 I, -0.167735 +
0.0310185 I, -0.149101 + 0.0828622 I,
0.52678 + 0.24822 I, -0.152329 - 0.0717778 I, -0.164023 -
0.0468378 I, -0.14054 - 0.0966731 I}, {0.0330804 +
0.125763 I, -0.153878 - 0.585007 I,
0.315782 - 0.422264 I, -0.48268 - 0.212239 I,
0.0655366 - 0.112319 I,
0.304854 - 0.522471 I, -0.191231 - 0.491382 I,
0.521878 - 0.0752944 I}, {0.00132524 - 0.009075 I,
0.0790595 - 0.541385 I, -0.279998 + 0.270281 I,
0.191015 + 0.339064 I,
0.00735408 - 0.00547991 I, -0.438721 + 0.326914 I,
0.104687 - 0.374822 I,
0.389106 + 0.00687122 I}, {0.478287 - 0.133411 I,
0.0843676 - 0.0235332 I, -0.342299 + 0.339145 I, -0.46842 -
0.113006 I, -0.432536 - 0.243864 I,
0.0762974 + 0.0430164 I, -0.251315 - 0.41113 I, -0.481853 -
0.00223001 I}, {-0.0639162 - 0.0690425 I,
0.296278 + 0.320041 I, -0.28542 + 0.251055 I,
0.272286 - 0.265243 I, 0.00362487 - 0.094016 I,
0.0168028 - 0.435803 I, -0.38009 - 0.00498043 I,
0.379345 + 0.0243003 I}, {-0.470095 + 0.186804 I, -0.347478 +
0.138079 I, -0.29015 + 0.303008 I, -0.418988 -
0.021215 I, -0.464498 - 0.200317 I, 0.343341 + 0.148068 I,
0.281268 + 0.31127 I,
0.419426 - 0.00909209 I}, {0.101492 + 0.272772 I, -0.241163 -
0.648157 I, -0.211764 + 0.174939 I, 0.274611 - 0.00602878 I,
0.121114 - 0.264644 I, 0.287789 - 0.628845 I,
0.198442 + 0.189916 I, -0.27344 -
0.0260391 I}}}, {{-9.42409, -7.44177, -4.12004, -2.51234,
3.70499, 4.06284, 7.05022,
7.10828}, {{-0.677397 + 0.0585892 I, -0.00400919 +
0.000346761 I, -0.0171565 - 0.0571935 I, -0.00708171 +
0.0592899 I, -0.520421 - 0.437563 I, 0.00308012 + 0.00258973 I,
0.0469318 - 0.0369168 I, -0.0283104 + 0.0525734 I}, {-0.523318 +
0.229985 I, -0.179851 + 0.0790397 I, -0.188072 +
0.0368023 I, -0.154295 + 0.113659 I,
0.532666 + 0.207418 I, -0.183063 - 0.0712842 I, -0.189472 -
0.0287341 I, -0.15901 - 0.106964 I}, {-0.0457089 - 0.147101 I,
0.175658 + 0.565305 I, -0.307154 + 0.434651 I,
0.499398 + 0.184031 I, -0.071695 + 0.136337 I, -0.275522 +
0.52394 I,
0.222998 + 0.483257 I, -0.524536 + 0.0901544 I}, {0.00117658 -
0.0069311 I, 0.0916398 - 0.539838 I, -0.273934 + 0.26512 I,
0.1711 + 0.340666 I,
0.005733 - 0.00406906 I, -0.446522 + 0.316924 I,
0.119902 - 0.361873 I,
0.381169 + 0.00623288 I}, {-0.480425 + 0.140868 I, -0.06232 +
0.0182731 I, 0.354126 - 0.341156 I, 0.48228 + 0.0959093 I,
0.43932 + 0.240103 I, -0.056988 - 0.0311459 I,
0.273205 + 0.408841 I,
0.491638 + 0.00917125 I}, {-0.0413165 - 0.0467935 I,
0.279035 + 0.316025 I, -0.29069 + 0.258302 I,
0.292314 - 0.256463 I, 0.00387288 - 0.0623032 I,
0.0261559 - 0.420771 I, -0.388044 - 0.0253503 I,
0.388196 + 0.0229022 I}, {0.487796 - 0.187144 I,
0.344936 - 0.132335 I, 0.296998 - 0.30563 I,
0.425208 + 0.028554 I,
0.477254 + 0.212593 I, -0.337482 - 0.150331 I, -0.280477 -
0.320858 I, -0.426122 + 0.00610367 I}, {0.101184 +
0.282508 I, -0.233844 - 0.652901 I, -0.211332 + 0.170614 I,
0.271597 - 0.00235316 I, 0.128216 - 0.271311 I,
0.296318 - 0.627024 I,
0.193712 + 0.190384 I, -0.270076 -
0.0287925 I}}}, {{-9.55153, -7.38605, -3.94069, -2.34127,
3.57007, 3.9788, 6.87615,
7.28141}, {{0.489182 - 0.489182 I, 0.00767275 - 0.00767275 I,
0.0533557 + 0.0265756 I, -0.0265756 - 0.0533557 I,
0.691807, -0.0108509, -0.0189364 + 0.05652 I, -0.0189364 -
0.05652 I}, {-0.396197 + 0.396197 I, -0.155242 +
0.155242 I, -0.181657 + 0.106054 I, -0.106054 + 0.181657 I,
0.560307, -0.219546, -0.203442 + 0.0534591 I, -0.203442 -
0.0534591 I}, {-0.0747996 + 0.157997 I, 0.246121 - 0.519874 I,
0.52566 - 0.103612 I, -0.253027 - 0.472262 I,
0.164612 - 0.0588292 I, 0.54164 - 0.193572 I,
0.155023 - 0.512856 I,
0.444962 + 0.298433 I}, {-0.00338552 + 0.00338552 I, -0.388792 +
0.388792 I, 0.367164 - 0.0666031 I,
0.0666031 - 0.367164 I, -0.00478784,
0.549835, -0.30672 + 0.212529 I, -0.30672 -
0.212529 I}, {-0.485554 + 0.147667 I, -0.0341219 + 0.0103772 I,
0.36826 - 0.343348 I, 0.497065 + 0.0801855 I,
0.447755 + 0.238922 I, -0.0314656 - 0.0167901 I,
0.294778 + 0.408178 I,
0.503183 + 0.0176153 I}, {-0.00276422 + 0.0352472 I,
0.0314966 - 0.40162 I,
0.399837 + 0.0213424 I, -0.398276 - 0.0412485 I, -0.0268782 +
0.022969 I, -0.30626 + 0.261717 I,
0.252456 + 0.31079 I, -0.267636 - 0.297819 I}, {-0.302329,
0.69608, -0.0903668 - 0.250349 I, -0.0903668 + 0.250349 I,
0.213779 + 0.213779 I,
0.492203 + 0.492203 I, -0.240923 + 0.113125 I,
0.113125 - 0.240923 I}, {0.491619 + 0.220697 I,
0.337305 + 0.151422 I, 0.431899 - 0.00337925 I,
0.284492 + 0.32498 I,
0.191571 + 0.503683 I, -0.131439 - 0.345582 I,
0.0286294 - 0.430962 I, -0.307788 -
0.303009 I}}}, {{-9.67576, -7.33705, -3.70994, -2.17392, 3.44493,
3.9318, 6.57736,
7.51153}, {{0.704577, 0.018062, 0.0255535 + 0.053998 I,
0.0255535 - 0.053998 I,
0.498211 + 0.498211 I, -0.0127718 - 0.0127718 I, -0.0562514 +
0.0201133 I, 0.0201133 - 0.0562514 I}, {0.549046, 0.236761,
0.217128 + 0.0610546 I,
0.217128 - 0.0610546 I, -0.388234 - 0.388234 I,
0.167415 + 0.167415 I, 0.196705 + 0.110361 I,
0.110361 + 0.196705 I}, {0.0713886 + 0.17741 I, -0.207166 -
0.514833 I, 0.290505 - 0.453105 I, -0.523374 - 0.125605 I,
0.0749682 - 0.175927 I,
0.217554 - 0.510531 I, -0.281265 - 0.458897 I,
0.525811 - 0.114976 I}, {0.0026574,
0.553823, -0.306294 - 0.198181 I, -0.306294 + 0.198181 I,
0.00187907 + 0.00187907 I, -0.391612 - 0.391612 I,
0.356718 + 0.0764478 I,
0.0764478 + 0.356718 I}, {-0.458365 - 0.240482 I,
0.000917614 + 0.000481428 I, 0.51634 + 0.027467 I,
0.31604 + 0.409243 I, 0.154066 + 0.494159 I,
0.00030843 + 0.000989272 I, -0.0659045 + 0.512853 I,
0.345686 + 0.38453 I}, {0.00944186 + 0.0078976 I, -0.291725 -
0.244012 I, 0.252309 - 0.328425 I, -0.278668 + 0.306377 I,
0.00109196 + 0.0122609 I, 0.0337381 + 0.378823 I,
0.413689 - 0.0195932 I, -0.410641 + 0.0538219 I}, {-0.297446,
0.699372, -0.0937752 - 0.240596 I, -0.0937752 + 0.240596 I,
0.210326 + 0.210326 I,
0.494531 + 0.494531 I, -0.236436 + 0.103818 I,
0.103818 - 0.236436 I}, {0.508109 - 0.225031 I,
0.341738 - 0.151349 I, 0.292996 - 0.324207 I,
0.436986 + 0.000913268 I,
0.518408 + 0.200166 I, -0.348665 - 0.134626 I, -0.30835 -
0.309641 I, -0.436429 +
0.0220696 I}}}, {{-9.79568, -7.29167, -3.43432, -2.01255,
3.32762, 3.92047, 6.23075,
7.73642}, {{0.50769 - 0.50769 I, 0.0179752 - 0.0179752 I,
0.0585483 + 0.0133341 I, -0.0133341 - 0.0585483 I,
0.717982, -0.0254208, -0.0319713 + 0.0508285 I, -0.0319713 -
0.0508285 I}, {0.53838, 0.248219, 0.227913 + 0.0644604 I,
0.227913 - 0.0644604 I, -0.380692 - 0.380692 I,
0.175518 + 0.175518 I, 0.206739 + 0.115579 I,
0.115579 + 0.206739 I}, {0.0825849 + 0.185118 I, -0.216398 -
0.485065 I, 0.284012 - 0.458976 I, -0.531269 - 0.0952617 I,
0.0725018 - 0.189294 I,
0.189977 - 0.496009 I, -0.308304 - 0.443024 I,
0.525372 - 0.123718 I}, {-0.000587409 +
0.000587409 I, -0.395476 + 0.395476 I, 0.345235 - 0.0873177 I,
0.0873177 - 0.345235 I, -0.000830722,
0.559288, -0.305861 + 0.182375 I, -0.305861 -
0.182375 I}, {-0.506692 + 0.160323 I, 0.0442719 - 0.0140081 I,
0.402599 - 0.34802 I, 0.529514 + 0.0530883 I,
0.471651 + 0.24492 I, 0.0412102 + 0.0213997 I,
0.336884 + 0.411962 I,
0.530768 + 0.0385935 I}, {0.000647736 - 0.00682218 I,
0.0334587 - 0.352399 I,
0.429224 + 0.0176433 I, -0.424875 - 0.0634496 I,
0.00528203 - 0.00436599 I, -0.272842 + 0.225525 I,
0.255566 + 0.345297 I, -0.291031 - 0.315983 I}, {-0.284855,
0.703035, -0.100754 - 0.226709 I, -0.100754 + 0.226709 I,
0.201423 + 0.201423 I,
0.497121 + 0.497121 I, -0.231552 + 0.0890637 I,
0.0890637 - 0.231552 I}, {-0.213247 + 0.532038 I, -0.141676 +
0.353472 I,
0.00887743 + 0.441362 I, -0.311255 + 0.313049 I, -0.526996 +
0.225419 I, 0.350123 - 0.149763 I, 0.44145 - 0.00126895 I,
0.305813 - 0.318367 I}}}}


The data format is a list of Eigenvalues and associated Eigenvectors for (in this example 6) different reciprocal lattice vectors of the band structure. For each of the 6 datapoints there is first a list of (in this case 8) Eigenvalues and second a list of 8 Eigenvectors. One can plot this data easily by applying

ListLinePlot[Transpose[data[[;; , 1]]]]


yielding

or showing the data points with

ListLinePlot[Transpose[data[[;; , 1]]]]


One can clearly see that the two top bands cross over which is not recognizable by the ListPlot routine since it's acting on the sorted Eigenvalues. If looking at the Eigenvectors in reality the two top bands are running from bottom left to top right and vice versa respectively and do not have the discontinuity in slope at their crossing. The characteristics of each Eigenvalue in the band can be determined by the associated (normalized) Eigenvector which has a characteristic combination of components. However these Eigenvectors also change over the running reciprocal vectors and thus such crossings are difficult to detect by just looking at the dedicated datapoints. Now the issue: I would need to sort the Energy Values in an order reflected by the Eigenvectors. E.g. by starting with the first point (the first reciprocal vector) in the dataset, take the canonical order of the Eigenvalues and the associated Eigenvectors as a reference and then use the differences of the Eigenvectors for the next point to determine the right order or Eigenvalues to prevent crossovers like shown in the example data. Thus for the above example the order of the Eigenvalues will stay canonical until the cross-over happens, wheras afterwards the two top Eigenvalues will get flipped. These cross-overs can be best seen by looking at the Norms of the complex components of the last two Eigenvectors of points #3,#4 and #5 of above data

Abs[data[[2, 2, -2 ;;]]]
Abs[data[[3, 2, -2 ;;]]]
Abs[data[[4, 2, -2 ;;]]]


yielding

{{0.505851, 0.373907, 0.419525, 0.419525, 0.505851, 0.373907, 0.419525, 0.419525},
{0.291042, 0.691569, 0.274677, 0.274677, 0.291042, 0.691569, 0.274677, 0.274677}}

{{0.522463, 0.36945, 0.426166, 0.426166, 0.522463, 0.36945, 0.426166, 0.426166},
{0.300081, 0.693515, 0.271607, 0.271607, 0.300081, 0.693515, 0.271607, 0.271607}}

{{0.302329, 0.69608, 0.26616, 0.26616, 0.302329, 0.69608, 0.26616, 0.26616},
{0.538884, 0.369734, 0.431912, 0.431912, 0.538884, 0.369734, 0.431912, 0.431912}}


The last vector-set indicates that the associated Eigenvalues need to get flipped to have the "natural" sorting.

Is there a simple built in way in Mathematica how to establish such an sorting?

• Have you seen this? Jul 4, 2016 at 6:00
• MapAt[Abs@#[[;; 2]] &, data, {All, 2, All}] // Map[Transpose] // Flatten[#, 1] & // Join[{#}, #2] & @@@ # & // ListPointPlot3D try this. In the meantime, could you explain a little about the structure of the eigenvectors? What do the 8 components mean? My snippet groups the data according to the values of the first two components. Aug 3, 2016 at 10:50

I've apparently figured out how to find which band is which by looking at the eigenvectors:

data // Map[Transpose] // Flatten[#, 1] & // (* structuring the data *)
Join[{#}, Abs@#2] & @@@ # & // (* construct a list of *)
(* eigenvalues catenated with their *)
(* eigenvectors Abs of each component *)
FindClusters[#, 8, DistanceFunction -> (Norm@*Rest@*Subtract)] & //
(* I shall find clusters by the
closeness of eigenvectors *)
#[[;; , ;; , ;; 3]] & // ListPointPlot3D (* display *)


Good, we can see that the eight bands are now separable. I need a FindClustersBy function... but that doesn't exist. I also lost the order of the eigenvalues in the process - now I don't know which one is the first, and which is the sixth.

So I'll roll my own FindClustersBy.

idxs = Table[{i, j}, {i, 6}, {j, 8}] // Flatten[#, 1] &;
FindClusters[idxs, 8,
DistanceFunction -> (Norm[
Abs[data[[First@#, 2, Last@#]]] -
Abs[data[[First@#2, 2, Last@#2]]]] &)]


Pay close attention to the last two rows of the table. Now I plot this:

Map[data[[First@#,1,Last@#]]&, %, {2}];
ListPlot@%


Now the two top bands correctly cross. If I knew more about the structure of the dataset, I could maybe explain, why I'm comparing the datapoints based on the absolute values of their eigenvector components, rather than the complex-valued vectors. There's probably an arbitrary phase there.

• @LLIAMNYP Very good solution. The comparison of the eigenvector component norms is perfectly o.k. since they describe the weighting factors of the linear combination of atomic orbitals (LCAO) and thus describe the "character" of the band. I'll have to check if the norm is sufficiently unique that the routine is not breaking down at some more complex bandstructure examples, but for the time being the solution looks very good. Aug 5, 2016 at 17:37
• @rainer you may run into problems if you check too many reciprocal space vectors at a time. Is there a possibility that two bands may have a very similar character at some point? Aug 5, 2016 at 17:53

This is how you can do what you are asking for.

np = 6 (*no of k points*)
ne = 8 (*no of bands*)
norm = Table[ Map[Norm[#] &, (data[[i, 2]] - data[[i - 1, 2]])], {i, 2, np}];
norm = Join[{Range[ne]}, norm];
(*norm contains Norm of difference between the eigen vectors.
you can use some other quantity as well  *)
data1 = Map[Transpose[Join[{data[[#, 1]]}, {norm[[#]]}]] &, Range[np]];
data2 = Transpose@Table[Sort[data1[[i]], #1[[2]] < #2[[2]] &][[All, 1]], {i, np}]
(*Eigenvaluse sorted according to norm values*)

ListLinePlot[data2]


• Your procedure of sorting does not resolve the issue I mentioned above. The norm in line 3 of your answer measures the differences of the unsorted list of Eigenvectors and thus for the case of the two topmost bands it sorts after the norm of the differences between two uncorrelated vectors in the case of the crossover. The issue is here that one would need to look at the multiple possible combinations of Eigenvectors for two adjacent k points. Jul 5, 2016 at 5:00
• @Rainer a more direct solution is given by keyword Nearest. You'd be looking for the nearest eigenvector in the eight eigenvectors for the k+1th point to each given eigenvector in the kth point. Aug 4, 2016 at 9:44