1
$\begingroup$

I have used the (apparently fastest) approach of

Do[MapThread[Prepend, {m, v}];

for prepending and appending v to a matrix m from question What is the most efficient way to add rows and columns to a matrix? but my matrix is quite large and it is too slow for my purposes. (Not at home now but rough performance guide is ~30 seconds to append a 5,000 row vector to a 5000 row matrix to 2 cols)

The specific context: a matrix of time series data (imported from CSV) of 50,000 rows (could be MUCH longer) and 2 columns (col 1 timestamp; col 2 value). I want to add a third column and re-export to CSV.

Is there a better approach than the method quoted?

$\endgroup$
  • $\begingroup$ mathematica.stackexchange.com/a/3077/21532 $\endgroup$ – yode Mar 24 '17 at 7:11
  • $\begingroup$ It is unclear from your description: do you actually use the imported data to create the new column? If not, there seems no need to even create the matrix. And even if you do need to actually import the data into a matrix, you need not create the bigger matrix to export the data. $\endgroup$ – Alan Mar 24 '17 at 14:17
  • $\begingroup$ Is the matrix packed? $\endgroup$ – J. M. will be back soon Mar 24 '17 at 17:04
  • $\begingroup$ @J.M.No, not packed. $\endgroup$ – Julian Moore Mar 25 '17 at 9:35
2
$\begingroup$

Adding the column shouldn't be too time consuming. Here's a 1,000,000 x 2 matrix of real numbers getting a column added, with two Transpose[ ] calls thrown in. Too cheap to meter. The file writing is the expensive part.

In[107]:= n = 1000000;

In[108]:= res = Table[RandomReal[20], {i, n}, {j, 2}];

In[109]:= vec = Table[RandomReal[20], {i, n}];

In[110]:= Timing[tres = Append[Transpose[res], vec] // Transpose;]

Out[110]= {0., Null}

In[111]:= Timing[Export["test.csv", tres]]

Out[111]= {19.2193, "test.csv"}
$\endgroup$
  • $\begingroup$ (Working to a deadline so no time to got back and investigate but to acknowledge and thank...) Yes, that seems fast enough ;) I think the problem may have been that one column was of reals and the other of DateObjects. I've bypassed the conversion of timestamps to DateObjects for now and with only text/numbers it is plenty fast. Thx again $\endgroup$ – Julian Moore Mar 25 '17 at 9:41
0
$\begingroup$
mat = RandomInteger[100, {10, 2}];  (* fake matrix *)
src = "/temp/temp99.csv";  (* location for fake data *)
Export[src, mat];
vec = RandomInteger[100, 10];  (* fake vector *)
tmp = StringTemplate["``,``"];  (* string template for conversion *)
fOld = OpenRead[src]; (* open your data file to read *)
fNew = OpenWrite["/temp/temp98.csv"] (* target for new data *)
(* each item of vec, read old line & write new line *)
Scan[WriteLine[fNew, tmp[ReadLine[fOld], #]] &, vec];
Close[fNew]; Close[fOld];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.