Formatting an imported data into Mathematica in a matrix form

Question

I have a file whose data read as {{0,:,1,2,3;},{1,:,0,164,165;},{2,:,0,3,167;},{3,:,0,2,166;},.., {"$$"}} I have to format this data into a matrix. There is also a first three line that I omitted for clarity. I was trying to remove the these lines by hand, but I found it cumbersome for each file. So, the reading of the data should be from the fourth line onward.

In this above data {{0,:,1,2,3;},.., {"$$"}}, the first entry tells us the lattice site, and after semi-colon :, it shows to which this lattice site is connected. It is one-dimensional data. Moreover, we should ignore the last part (or last line) in the data, i.e. {"$$"}. Essentially, the first-three and the last lines are to be ignored while reading the data. The truncated data can be found here pastebin.

For instance, in this entry {{0,:,1,2,3;}, the lattice site 0 is connected to the lattice site 1, 2, and 3, and so on and so forth. Hence the matrix entry coming from this case would look like {{0,1,1,1,...,0},...}
The total lattice sites in the above case is 165, so the matrix would be 165 $\times$ 165

My MWE:

data=Import["file.dat"];
data[[4,2]]=1;
data[[4,3]]=1;
data[[4,4]]=1;

Unfortunately, I have no idea how to proceed further. I found myself very ignorant. I searched through the Mathematica.stack but could not follow the answers. I have never done this. Even if a full answer is not given but a hint, I will try to reconstruct the answer or MWE—apologies for the ignorance.

Answer 1

165 × 165 cannot be correct since indexes start at 0 and the last index is 167 so it has to be 168 x 168.

rawData = Import["~/Downloads/file.dat", "Data"];
data = Take[rawData, {4, -2}] (* Ignore first 3 and last line *)

(* Function to convert site spec like "{0,:,1,2,3;}" to array *)
expandRow[spec_] := Module[
  {specList = (spec // DeleteCases[":"]) /. 
     x_?StringQ :> ToExpression@StringTake[x, 1 ;; -2]},
  SparseArray[Thread[Rest[specList + 1] -> 1], 168] // Normal]

Then map it over the data

expandRow /@ data

Answer 2

you must adapt the code to different forms of input. Here is code that reads the new file:

(*input=StringToStream[str];*)
input = OpenRead["FileName"];
line = ReadLine[input];
n = StringCases[line, NumberString][[1]] // ToExpression;;

dat = Reap[
    Do[
     line = Read[input, "String"];
     Sow[StringCases[line, NumberString] // ToExpression];
     , {i, n}]
    ][[2, 1]];

Close[input];
dat += 1;
n = Max[dat];
mat = ConstantArray[0, {n, n}];
Scan[Function[x, (mat[[x[[1]], #]] = 1) & /@ Rest[x]], dat];
mat

Answer 3

I assume you will import your matrix as a string, since : is not allowed as an entry in a Mathematica list. If you do, you can use string operations to single out the commands, transform the string to normal Mathematica expressions and use them as you want. For instance, consider the following simple procedure (careful, I use s = s+1 to shift to indices / lattice sites beginning from 1 instead of 0 as in your example)

n = 7;
matrix = ConstantArray[0, {n, n}];
strings = "{{0,:,1,3,6;},{2,:,1,0,5,3;},{4,:,2,0;}}";
strings = StringSplit[strings, {",{"}];
Do[
 s = StringDelete[s, {"{{", ";}}", ";}", ":"}];
 s = StringSplit[s, ","];
 s = ToExpression@{s[[1]], s[[3 ;;]]};
 s = s + 1;
 Print@InputForm@s;
 matrix[[s[[1]], s[[2]]]] = 1;
 , {s, strings}
 ]
MatrixForm@matrix
smatrix = SparseArray@matrix;
MatrixForm@Normal@smatrix

Since your matrices will be large, consider using SparseArray (take a look into the documentation, I am sure you will find out how to change the code above for that).

Answer 4

For an example, we will read the data from a string. If you want to read from a file, activate the second statement and inactivate the first one:

str = "
!Hello 1.
n=11 $=0 g
0:1 2 3;
1:0 164 165;
2:0 3 167;
3:0 2 166;
4:5 6 107;
167:2 37 44;
$$";

input = StringToStream[str];
(*input=OpenRead[FileName];*)

Then we read away the first 3 lines:

Do[Read[input, "String", NullRecords -> True], 3];

Next we read all the data and close the input stream:

line = Read[input, "String"];
dat = Reap[
   While[line =!= "$$",
    Sow[ReadList[StringToStream[line], Number, 10^4, 
      RecordSeparators -> {" ", ":", ";"}]];
    line = Read[input, "String"];
    ]][[2, 1]];
Close[input];

As the data is zero based, we need to add 1;

dat += 1;

Then we create a matrix of zeros with a dimension that corresponds to the largest data value:

n = Max[dat];
mat = ConstantArray[0, {n, n}];

As far, the data is contained in strings that we need to interpret and set the corresponding matrix elements to one:

Scan[Function[x, (mat[[x[[1]], #]] = 1) & /@ Rest[x]], dat];

The result is contained in the variable mat, that is too large to be displayed in full, so we only show the beginning:

For convenience, here is the full code:

input = StringToStream[str];
(*input=OpenRead[FileName];*)
Do[Read[input, "String", NullRecords -> True], 3]

line = Read[input, "String"];
dat = Reap[
   While[line =!= "$$",
    Sow[ReadList[StringToStream[line], Number, 10^4, 
      RecordSeparators -> {" ", ":", ";"}]];
    line = Read[input, "String"];
    ]][[2, 1]];
Close[input];
dat += 1;
n = Max[dat];
mat = ConstantArray[0, {n, n}];
Scan[Function[x, (mat[[x[[1]], #]] = 1) & /@ Rest[x]], dat];
mat