Storing result from nested Table in a File for further computations

Question

Consider the following:

 Table[Table[x = 20; y = 30; 2 i x + 3 j y , {i, 10}], {j, 3}]

Suppose I want the result of this nested Table to be stored to an appendable file, I could do something like:

Table[Table[x = 20; y = 30; 2 i x + 3 j y >> test.dat, {i, 10}], {j, 3}]

Where test.dat has been previously opened using OpenAppend["test.dat"]

However, this doesn't work as I would like it to as the answer is stored differently than the actual output shown below and Mathematica returns Null for every element of the output.

{ 
    {130, 170, 210, 250, 290, 330, 370, 410, 450, 490}, {220, 260, 300, 
     340, 380, 420, 460, 500, 540, 580}, {310, 350, 390, 430, 470, 510, 
     550, 590, 630, 670} 
  }

My question is this. How can I save the output of the Nested Table to a file for every iteration of the outer Table and prevent any List from being returned by Mathematica?. That is, I don't want Mathematica to return the following List as it currently does:

{ {Null, Null, Null, Null, Null, Null, Null, Null, Null, Null}, {Null, 
  Null, Null, Null, Null, Null, Null, Null, Null, Null}, {Null, Null, 
  Null, Null, Null, Null, Null, Null, Null, Null} }

I will also like the data stored in the file to be exactly as the output shown above, since I will use it for further calculations in Mathematica. This is basically a way to prevent excessive memory usage. If there are better ways to achieve the same thing please feel free to post that as an answer.

Answer 1

Since you state outer Table:

How can I save the output of the Nested Table to a file for every iteration of the outer Table and prevent any List from being returned by Mathematica?

It seems to me that you want something like this (note >>>):

Do[Table[x = 20; y = 30; 2 i x + 3 j y, {i, 10}] >>> "accum.dat", {j, 3}]

Which produces a file with contents:

{130, 170, 210, 250, 290, 330, 370, 410, 450, 490}
{220, 260, 300, 340, 380, 420, 460, 500, 540, 580}
{310, 350, 390, 430, 470, 510, 550, 590, 630, 670}

To read this file as a matrix just use: ReadList["accum.dat"]

The difference compared to the last line of AugR's answer (which I voted for) is that this outputs each row as it is created rather than at the end, which would be useful if memory is a concern or if calculation may be interrupted and you want to save results as you go.

Answer 2

This is not really idiomatic Mathematica, but the following snippet is probably close to what you have in mind. Note that Oleksandr's comment about Do is a good one. Also, the reason that I've put manual writing of certain braces and commas in here is because I'm assuming you want to be able to read the data back in (as a whole list, potentially read from a file with many other expressions in it) with Get/Read/ReadList etc.

(* Open the stream in any way you'd like. *)
stream = OpenAppend["test.dat"];

(* Write an opening brace to begin the list which you intend to \
manually construct as it is generated... *)
WriteString[stream, "\n{\n"];

(* Compute your memory-intensive nested tables. *)
iMax = 10; jMax = 3;
Do[
  (* Write each nested table... *)
  WriteString[
   stream,
   Table[x = 20; y = 30; 2 i x + 3 j y, {i, iMax}]
   ];
  (* Write a terminating comma, if necessary... *)
  WriteString[
   stream,
   If[j == jMax, "\n", ",\n"]
   ],
  {j, jMax}
  ];

(* After the computation, write a terminating brace. *)
WriteString[stream, "}\n"];
Close[stream];

(* Check it. *)
FilePrint["test.dat"]
Read["test.dat"]
Close["test.dat"];

(* Printed file:
{
{130, 170, 210, 250, 290, 330, 370, 410, 450, 490},
{220, 260, 300, 340, 380, 420, 460, 500, 540, 580},
{310, 350, 390, 430, 470, 510, 550, 590, 630, 670}
}
*)

(* Data read back in:
{{130, 170, 210, 250, 290, 330, 370, 410, 450, 490}, {220, 260, 300, 
  340, 380, 420, 460, 500, 540, 580}, {310, 350, 390, 430, 470, 510, 
  550, 590, 630, 670}}
*)

Finally, unless memory is really a major issue and you can't afford to have all of your tables in memory at once, the preferred way to do this would simply be:

PutAppend[Table[x = 20; y = 30; 2 i x + 3 j y, {j, 3}, {i, 10}], "test.dat"]