Fast way to add a column to a large matrix

Question

I have used the (apparently fastest) approach of

Do[MapThread[Prepend, {m, v}];

for prepending and appending v to a matrix m from question What is the most efficient way to add rows and columns to a matrix? but my matrix is quite large and it is too slow for my purposes. (Not at home now but rough performance guide is ~30 seconds to append a 5,000 row vector to a 5000 row matrix to 2 cols)

The specific context: a matrix of time series data (imported from CSV) of 50,000 rows (could be MUCH longer) and 2 columns (col 1 timestamp; col 2 value). I want to add a third column and re-export to CSV.

Is there a better approach than the method quoted?

Answer 1

Adding the column shouldn't be too time consuming. Here's a 1,000,000 x 2 matrix of real numbers getting a column added, with two Transpose[ ] calls thrown in. Too cheap to meter. The file writing is the expensive part.

In[107]:= n = 1000000;

In[108]:= res = Table[RandomReal[20], {i, n}, {j, 2}];

In[109]:= vec = Table[RandomReal[20], {i, n}];

In[110]:= Timing[tres = Append[Transpose[res], vec] // Transpose;]

Out[110]= {0., Null}

In[111]:= Timing[Export["test.csv", tres]]

Out[111]= {19.2193, "test.csv"}

Answer 2

mat = RandomInteger[100, {10, 2}];  (* fake matrix *)
src = "/temp/temp99.csv";  (* location for fake data *)
Export[src, mat];
vec = RandomInteger[100, 10];  (* fake vector *)
tmp = StringTemplate["``,``"];  (* string template for conversion *)
fOld = OpenRead[src]; (* open your data file to read *)
fNew = OpenWrite["/temp/temp98.csv"] (* target for new data *)
(* each item of vec, read old line & write new line *)
Scan[WriteLine[fNew, tmp[ReadLine[fOld], #]] &, vec];
Close[fNew]; Close[fOld];