Import a partitioned lower triangular matrix

Question

The output of a program that I use gives me a lower triangular matrix that is separated into multiple sections of four columns, with the number of rows consistently decreasing.

A small version of such a matrix output looks like this:

======  SO matrix real part

 column           1                     2                     3                     4
 row
    1    3.70130204704439E-01
    2   -9.63836222573747E-05  2.20929173706067E-01
    3   -2.69997763259066E-04 -8.35046872617608E-06  4.33952219457710E-01
    4   -1.61420894543695E-05  4.73806714804249E-05  1.76569354148931E-05  4.91757486427508E-01
    5   -2.58965512742325E-06 -2.47507081676155E-06 -2.47223559300728E-07 -2.23697908682022E-06
    6   -2.01946256051423E-37 -4.05215757045853E-22 -1.07418624224282E-21 -1.13156356527472E-22

 column           5                     6
 row
    5    5.38033500671857E-01
    6    4.35830155705519E-23  3.22565122194430E-01

(You can obtain the bigger matrix here.)

Now, I want to import it from the program's output into Mathematica to give me either a "normal" lower triangular matrix, or a square matrix with the missing upper triangular matrix values to be zero.

(For a bigger matrix of 40 excited states,) I came up with something that is very impractical on a more routine basis... admittedly my first try.

file = Import["/path/to/file.out", "Table"];
(* get first four important rows (80x4) *)
rpart1 = Select[
    Select[file[[
      9641 ;; 10539]], # != {} &], #[[1]] != "column" && #[[1]] != 
       "row" &][[1 ;; 80, 2 ;;]];
(* get second four important rows (76x4) *)
rpart2 = Select[
    Select[file[[
      9641 ;; 10539]], # != {} &], #[[1]] != "column" && #[[1]] != 
       "row" &][[81 ;; 81 + 75, 2 ;;]];
(* get the third four important rows (72x4) *)
rpart3 = Select[
    Select[file[[
      9641 ;; 10539]], # != {} &], #[[1]] != "column" && #[[1]] != 
       "row" &][[82 + 75 ;; 82 + 75 + 71, 2 ;;]];
(* create a square 12x12 matrix *)
rarr = ConstantArray[0, {Length[rpart1], 12}];
(* replace the matrix elements with the parsed data *)
Do[rarr[[i, j]] = rpart1[[i, j]], {i, 1, 4}, {j, 1, i}];
Do[rarr[[i, j]] = rpart1[[i, j]], {i, 5, Length[rpart1]}, {j, 1, 4}];
Do[rarr[[i + 4, j + 4]] = rpart2[[i, j]], {i, 1, 4}, {j, 1, i}];
Do[rarr[[i + 4, j + 4]] = rpart2[[i, j]], {i, 5, Length[rpart2]}, {j, 
   1, 4}];
Do[rarr[[i + 8, j + 8]] = rpart3[[i, j]], {i, 1, 4}, {j, 1, i}];
Do[rarr[[i + 8, j + 8]] = rpart3[[i, j]], {i, 5, Length[rpart3]}, {j, 
   1, 4}];
RealPartTriplets = rarr[[21 ;; 51, ;; 10]];

Now what would be a more dynamic/smarter way to import the data?

Answer 1

This works on both your small case and the one in Pastebin:

raw = Import["https://pastebin.com/raw/zUs5n8vz", "Table", "HeaderLines" -> 4];

matrix = SparseArray[PadRight[MapThread[Join,
                     PadLeft[Drop[Cases[#, {__?NumberQ}], None, 1] & /@
                             Split[raw, # =!= {} &], Automatic, {{}}]], Automatic, 0.]];

MatrixPlot[matrix]

Answer 2

You may make use of the positions of "column" to get the locations of each matrix part. Then use the row indicators on each line to build the matrix rows. Finally the resulting ragged list can be padded to the desired dimensions.

ClearAll[buildMatrix];
buildMatrix[tab_List] :=
 Module[{pos, res},
  pos = Plus[{2, -1}, #] & /@ Partition[Position[tab, "column"][[All, 1]], 2, 1, {1, 1}, 0];
  res = tab[[Span @@ pos[[1]]]] /. {} -> Nothing;
  (res[[First@#]] = Join[res[[First@#]], Rest@#]) & /@ 
     (tab[[Span @@ pos[[#]]]] /. {} -> Nothing) & /@ 
     Range[2, Length@pos];
  PadRight[res[[;; -2, 2 ;;]], ConstantArray[Max[Length /@ res - 1], 2]]
  ]

buildMatrix performs these steps on the list returned by a "Table" Import. There are several functions in it. To assist your understanding I suggest you read all of the Tutorials listed in the Elements of Lists guide of the documentation.

With

file = Import["/path/to/file.out", "Table"];

Then

buildMatrix[file] // MatrixForm

Hope this helps.

Answer 3

I guess there is still a lot to improve but at least, it’s now possible to use it routinely.

At first, the file is loaded with lines separated

file = Import["/path/to/file.out", "Lines"];

Then I search for the number of excitations, which also could be given by hand

nexc = ToExpression[
  Last[StringSplit[
    file[[Position[file, 
        "*   Final excitation energies with spin-orbit coupling \
effect            *"][[1, 1]] + 4]]]]]

I also search the position where the matrix starts

RealPos = Position[file, " ======  SO matrix real part"][[1, 1]]

How many lines do I have to read in?

linesToRead = 
 With[{range = Table[Range[4 n + 1, nexc], {n, 0, nexc/4 - 1}]}, 
  Length@Flatten@range + 3 Length@range]

As the values are in E-scientific notation and they don’t get properly converted when I do ToExpression[SplitString[...]], I load the file again but this time as a "Table".

file = Import["/path/to/file.out", "Table"];

Now I can read matrix parts

matrixParts = With[
   {stuff = 
     Select[file[[RealPos + 3 ;; 
         RealPos + linesToRead + 2]], # != {} && 
        First[#] != "column" && First[#] != "row" &]},
   Map[stuff[[#]] &, 
    Range @@@ 
     Table[{Sum[nexc - 4 m, {m, 0, n - 1}] + 1, 
       Sum[nexc - 4 m, {m, 0, n}]}, {n, 0, nexc/4 - 1}]]
   ];

I create a square matrix of the wanted size:

arr = ConstantArray[0, {nexc, nexc}];

and put the values at their corresponding positions:

Do[Part[arr, 1 + 4 n ;; nexc, 4 n + 1 ;; 4 n + 4] = 
  Map[PadRight[#, 5] &, matrixParts[[n + 1]]][[All, 2 ;;]], {n, 0, 
  nexc/4 - 1}]

---

As for my nexc/4-1 parts in the tables, it will not work with less than 4 rows/excited states.