# Import a partitioned lower triangular matrix

The output of a program that I use gives me a lower triangular matrix that is separated into multiple sections of four columns, with the number of rows consistently decreasing.

A small version of such a matrix output looks like this:

======  SO matrix real part

column           1                     2                     3                     4
row
1    3.70130204704439E-01
2   -9.63836222573747E-05  2.20929173706067E-01
3   -2.69997763259066E-04 -8.35046872617608E-06  4.33952219457710E-01
4   -1.61420894543695E-05  4.73806714804249E-05  1.76569354148931E-05  4.91757486427508E-01
5   -2.58965512742325E-06 -2.47507081676155E-06 -2.47223559300728E-07 -2.23697908682022E-06
6   -2.01946256051423E-37 -4.05215757045853E-22 -1.07418624224282E-21 -1.13156356527472E-22

column           5                     6
row
5    5.38033500671857E-01
6    4.35830155705519E-23  3.22565122194430E-01


(You can obtain the bigger matrix here.)

Now, I want to import it from the program's output into Mathematica to give me either a "normal" lower triangular matrix, or a square matrix with the missing upper triangular matrix values to be zero.

(For a bigger matrix of 40 excited states,) I came up with something that is very impractical on a more routine basis... admittedly my first try.

file = Import["/path/to/file.out", "Table"];
(* get first four important rows (80x4) *)
rpart1 = Select[
Select[file[[
9641 ;; 10539]], # != {} &], #[] != "column" && #[] !=
"row" &][[1 ;; 80, 2 ;;]];
(* get second four important rows (76x4) *)
rpart2 = Select[
Select[file[[
9641 ;; 10539]], # != {} &], #[] != "column" && #[] !=
"row" &][[81 ;; 81 + 75, 2 ;;]];
(* get the third four important rows (72x4) *)
rpart3 = Select[
Select[file[[
9641 ;; 10539]], # != {} &], #[] != "column" && #[] !=
"row" &][[82 + 75 ;; 82 + 75 + 71, 2 ;;]];
(* create a square 12x12 matrix *)
rarr = ConstantArray[0, {Length[rpart1], 12}];
(* replace the matrix elements with the parsed data *)
Do[rarr[[i, j]] = rpart1[[i, j]], {i, 1, 4}, {j, 1, i}];
Do[rarr[[i, j]] = rpart1[[i, j]], {i, 5, Length[rpart1]}, {j, 1, 4}];
Do[rarr[[i + 4, j + 4]] = rpart2[[i, j]], {i, 1, 4}, {j, 1, i}];
Do[rarr[[i + 4, j + 4]] = rpart2[[i, j]], {i, 5, Length[rpart2]}, {j,
1, 4}];
Do[rarr[[i + 8, j + 8]] = rpart3[[i, j]], {i, 1, 4}, {j, 1, i}];
Do[rarr[[i + 8, j + 8]] = rpart3[[i, j]], {i, 5, Length[rpart3]}, {j,
1, 4}];
RealPartTriplets = rarr[[21 ;; 51, ;; 10]];


Now what would be a more dynamic/smarter way to import the data?

This works on both your small case and the one in Pastebin:

raw = Import["https://pastebin.com/raw/zUs5n8vz", "Table", "HeaderLines" -> 4];

PadLeft[Drop[Cases[#, {__?NumberQ}], None, 1] & /@
Split[raw, # =!= {} &], Automatic, {{}}]], Automatic, 0.]];

MatrixPlot[matrix] • Very nice! (+1) – Edmund Aug 14 '17 at 13:16
• I already had code after this question got bumped, but I wanted to wait for OP to post his larger matrix to be sure mine worked. :) Also, PadLeft[]/PadRight[] with Automatic is really handy for ragged arrays. – J. M. will be back soon Aug 14 '17 at 14:08

You may make use of the positions of "column" to get the locations of each matrix part. Then use the row indicators on each line to build the matrix rows. Finally the resulting ragged list can be padded to the desired dimensions.

ClearAll[buildMatrix];
buildMatrix[tab_List] :=
Module[{pos, res},
pos = Plus[{2, -1}, #] & /@ Partition[Position[tab, "column"][[All, 1]], 2, 1, {1, 1}, 0];
res = tab[[Span @@ pos[]]] /. {} -> Nothing;
(res[[First@#]] = Join[res[[First@#]], Rest@#]) & /@
(tab[[Span @@ pos[[#]]]] /. {} -> Nothing) & /@
Range[2, Length@pos];
PadRight[res[[;; -2, 2 ;;]], ConstantArray[Max[Length /@ res - 1], 2]]
]


buildMatrix performs these steps on the list returned by a "Table" Import. There are several functions in it. To assist your understanding I suggest you read all of the Tutorials listed in the Elements of Lists guide of the documentation.

With

file = Import["/path/to/file.out", "Table"];


Then

buildMatrix[file] // MatrixForm Hope this helps.

• Thank you very much! When I use it on a bigger matrix (see comment on question), I get a lot of error messages. But in the end, the result seems ok. – pH13 - Yet another Philipp Aug 14 '17 at 9:50
• @pH13-YetanotherPhilipp May you provide a link to one of the bigger matrices. I'm curious as to why it errors. If the file format is the same then it should not complain. Also, what version are you using? – Edmund Aug 14 '17 at 10:09
• The link is commented below my question as of J.M.’s request. I use MMA 11.0.0.0. – pH13 - Yet another Philipp Aug 14 '17 at 10:11
• @pH13-YetanotherPhilipp See update. There is an empty gap row that is there for all but the last set of rows. This needs to be removed before updating the rows to get rid of the errors. – Edmund Aug 14 '17 at 10:23

I guess there is still a lot to improve but at least, it’s now possible to use it routinely.

At first, the file is loaded with lines separated

file = Import["/path/to/file.out", "Lines"];


Then I search for the number of excitations, which also could be given by hand

nexc = ToExpression[
Last[StringSplit[
file[[Position[file,
"*   Final excitation energies with spin-orbit coupling \
effect            *"][[1, 1]] + 4]]]]]


I also search the position where the matrix starts

RealPos = Position[file, " ======  SO matrix real part"][[1, 1]]


How many lines do I have to read in?

linesToRead =
With[{range = Table[Range[4 n + 1, nexc], {n, 0, nexc/4 - 1}]},
Length@Flatten@range + 3 Length@range]


As the values are in E-scientific notation and they don’t get properly converted when I do ToExpression[SplitString[...]], I load the file again but this time as a "Table".

file = Import["/path/to/file.out", "Table"];


Now I can read matrix parts

matrixParts = With[
{stuff =
Select[file[[RealPos + 3 ;;
RealPos + linesToRead + 2]], # != {} &&
First[#] != "column" && First[#] != "row" &]},
Map[stuff[[#]] &,
Range @@@
Table[{Sum[nexc - 4 m, {m, 0, n - 1}] + 1,
Sum[nexc - 4 m, {m, 0, n}]}, {n, 0, nexc/4 - 1}]]
];


I create a square matrix of the wanted size:

arr = ConstantArray[0, {nexc, nexc}];


and put the values at their corresponding positions:

Do[Part[arr, 1 + 4 n ;; nexc, 4 n + 1 ;; 4 n + 4] =
Map[PadRight[#, 5] &, matrixParts[[n + 1]]][[All, 2 ;;]], {n, 0,
nexc/4 - 1}]


As for my nexc/4-1 parts in the tables, it will not work with less than 4 rows/excited states.