# Memory Issues Processing Temporal Data

I'm doing some basic trade accounting using MMA, as follows:

TradeMarkToMarket[tradedataset_, tsMonthEndPrices_] :=
Module[{trdataset = tradedataset, tsprices = tsMonthEndPrices,
For[i = 1, i <= ntrades, i++,
"Entry Contracts"], tsprices,
If[Echo[i] == 1,


The program runs through a dataset of trade data, calculating monthly mark-to-market profits on each trade and these time series are combined in a Temporal Data variable tdTradeM2. All very routine.

However, I am finding that the processing speed quickly becomes extremely slow. To process the first 100 entries in the trade dataset takes approximately 4 seconds. Processing the first 1,000 entries times out after several hours.

I assume that the reason is the inefficiency of handling temporal data in-memory as I have done here. There are plenty of easy work-arounds. However, I am surprised that a work-around should be necessary, for a dataset this small.

Am I missing something? Perhaps if I pre-specified the size of the Temporal Data variable, for instance.

Suggestions to improve the efficiency of the code would be very welcome.

• Please, desribed your hardware. Also, there is a bracket missing in your code. And. please, add a sample dataset. – Jose Enrique Calderon Dec 25 '16 at 14:35
• Corrected the bracket. But the hardware configuration is irrelevant, as is the structure of the dataset: the key question is why the processing time doesn't appear to scale linearly (or even quadratically) with time. This cannot be to do with the hardware configuration or structure of the dataset. Rather, it must be to do with the way in which MMA handles Temporal Data objects of unspecified size. – Jonathan Kinlay Dec 25 '16 at 17:33

The code, as written, has a quadratic complexity, due to immutability of Mathematica expressions.

Suppose at step $k$ the temporal data already contains $k$ time series. To add another one, it create a new time series with $k+1$ pathes, where it copies $k$ previous paths, and the new path. Simple calculations show that the total numbers of copies is $n (n+1)/2$, where $n$ is the number of iterations.

This issue is not specific to time-series. It can be reproduced with lists as well. The following code also suffers from the quadratic complexity, for the same reasons:

quadratic[n_Integer?Positive] := Module[{li={}},
Do[li = Append[li, k] ,{k,1,n}];
li
]


Timing:

In[66]:= perfData =
Table[{n,
AbsoluteTiming[quadratic[n];] // First}, {n, {5000, 7000, 10000,
12000, 15000, 20000, 25000, 30000}}]

Out[66]= {{5000, 0.0943531}, {7000, 0.175722}, {10000,
0.297438}, {12000, 0.476399}, {15000, 0.752801}, {20000,
1.39099}, {25000, 2.06339}, {30000, 3.38593}}


The solution to this performance problem is to create nested lists at each iteration, and flatten at the end:

better[n_Integer?Positive] :=
Module[{li = {}},
Do[li = {li, k}, {k, 1, n}];
Flatten[li]
]


Timing:

In[71]:= perfData2 =
Table[{n,
AbsoluteTiming[better[n];] // First}, {n, {5000, 7000, 10000,
12000, 15000, 20000, 25000, 30000}}]

Out[71]= {{5000, 0.00376836}, {7000, 0.00716328}, {10000,
0.0100032}, {12000, 0.0146151}, {15000, 0.0145891}, {20000,
0.0270078}, {25000, 0.0243487}, {30000, 0.0235453}}


The same solution can be applied to the problem at hand.

Here is the condensed reproducer of your problem, which takes long time indeed:

In[77]:= (tdOut = Block[{ts, td, i},
Do[
ts =
TemporalData[
RandomReal[1,
10], {0 + FoldList[Plus, 0, RandomInteger[{1, 3}, 9]]}];
If[i == 1, td = ts,
td = TemporalData[{td, ts}, ResamplingMethod -> None]]
, {i, 1, 1000}];
td
];) // AbsoluteTiming

Out[77]= {41.1895, Null}


Applying the same method significantly improves performance:

In[78]:= (tdOut = Block[{ts, td = {}, i},
Do[
ts =
TemporalData[
RandomReal[1,
10], {0 + FoldList[Plus, 0, RandomInteger[{1, 3}, 9]]}];
td = {td, ts};
, {i, 1, 1000}];
TemporalData[Flatten[td], ResamplingMethod -> None]
];) // AbsoluteTiming

Out[78]= {0.906415, Null}