# Speed up compoundReturn across an array

I am trying to calculate a moving-window of compounded returns on a large 2D-array. (ie some kind of MovingMap on geometric returns)

The Rows are time-steps and the Columns are different financial instruments. The array contains returns at that time-step.

I want to get back a 2D array of compounded returns that has the same size/dimensions as the 2D array input.

The returns array I am working with has some Missing[] data as not all instruments start at the same time

The method I have come up with doesn't feel as fast as it can be and I would like to optimize it.

Before we start, this is an example of calculating compound returns on a single financial instrument.

compoundReturns[returns_List, window_Integer] :=
Module[
{
movingWindow = Partition[returns + 1., window, 1],
},
];


So

window = 3;
returns = {0.0634779, -0.0624394, 0.00844932,
0.0400948, -0.0154299};
compoundReturns[returns, 3]


{0, 0, 0.00549958, -0.0166087, 0.0326987}

I would like to perform this calculation on a list of financial instruments, preferably in parallel or threaded.

Let's make some small data to work with:

SeedRandom[1];
$$rows = 5;$$cols = 3;
$$return = RandomReal[{-0.1, 0.1}, {rows, cols}]; (*make data with missing*)$$return[[2, 2]] = Missing[];
$$return[[1, 2 ;; 3]] = Missing[];$$return


{{0.0634779, Missing[], Missing[]}, {-0.0624394, Missing[], -0.0868522}, {0.00844932, -0.0537691, -0.0207988},
{0.0400948, -0.0576348, 0.0497314}, {-0.0154299, -0.050501, 0.0954344}}

Given that this type of problem is entirely numerical my first instinct is to Compile it. I convert the Missing[] data in $return to 0. so that it can be handled by Compile. $$returnWithZero =$$return /. Missing[] -> 0.;  I keep track of when the data starts $$dataStart = Flatten[FirstPosition[#, _Real] & /@ (Transpose@$$return)]  {1, 3, 2} compoundReturnArrayC = Compile[ { {array, _Real, 2}, {window, _Integer}, {startRows, _Integer, 1} }, With[ { $$totalColumns = Length[startRows],$$totalRows = Length[array], returnArrayByInstruments = Transpose[array] + 1. }, Module[ { returnsByInstrument, returnsByInstrumentShortened, windows, compoundedReturns, compoundedReturnsByInstrument, startRow } , compoundedReturnsByInstrument = Table[ returnsByInstrument = returnArrayByInstruments[[c]]; startRow = startRows[[c]]; returnsByInstrumentShortened = returnsByInstrument[[startRow ;; $$totalRows]]; windows = Partition[returnsByInstrumentShortened, window, 1]; compoundedReturns = Fold[Times, #] & /@ windows; PadLeft[compoundedReturns - 1.,$$totalRows] , {c,$totalColumns}
];
Transpose[compoundedReturnsByInstrument]
]
]
,
CompilationTarget -> "C",
Parallelization -> True
];


So in our small example we get the correct result

compoundReturnArrayC[$$returnWithZero, window,$$dataStart]


{{0., 0., 0.}, {0., 0., 0.}, {0.00549962, 0., 0.}, {-0.0166087, 0., -0.061377}, {0.0326987, -0.153336, 0.125995}}

However for large arrays, increasing the window size slows down the code quite a bit.

SeedRandom[1];
$$rows = 10000;$$cols = 100;
$$return = RandomReal[{-0.1, 0.1}, {rows,$$cols}];

$$return[[2, 2]] = Missing[];$$return[[1, 2 ;; 3]] = Missing[];

$$dataStart = Flatten[FirstPosition[#, _Real] & /@ (Transpose@$$return)];
$$returnWithZero =$$return /. Missing[] -> 0.;

compoundReturnArrayC[$$returnWithZero, 500,$$dataStart]; // RepeatedTiming (*{3.19958, Null}*)
compoundReturnArrayC[$$returnWithZero, 5,$$dataStart]; // RepeatedTiming (*{0.209141, Null}*)


How can I increase the speed of this pretty trivial calculation? I feel there is some way to operate on all the lists in parallel, and then padding the final outcome with zeroes in the correct place, instead of looping through with table.

Edit

For reference, the fastest ways I have found to do a moving-map of Times to a list:

SeedRandom[1];
rng = RandomReal[{0.95, 1.05}, 10000];
window = 1000;

a = MovingMap[Apply[Times], rng, Quantity[window, "Events"]]; // RepeatedTiming
b = Times @@@ Partition[rng, window, 1]; // RepeatedTiming
c = Apply[Times][Transpose@Partition[rng, window, 1]]; // RepeatedTiming
d = Fold[Times, Transpose@Partition[rng, window, 1]]; // RepeatedTiming


{0.679459, Null}

{0.858843, Null}

{0.12836, Null}

{0.144564, Null}

SeedRandom[1];
rng = RandomReal[{0.95, 1.05}, 10000];
window = 1000;

a = MovingMap[Apply[Times], rng,
Quantity[window, "Events"]]; // RepeatedTiming
b = Times @@@ Partition[rng, window, 1]; // RepeatedTiming
c = Apply[Times][
Transpose@Partition[rng, window, 1]]; // RepeatedTiming
d = Fold[Times, Transpose@Partition[rng, window, 1]]; // RepeatedTiming
e = Exp[window MovingAverage[Log@rng, window]]; // RepeatedTiming
f = Exp@ListConvolve[ConstantArray[1., window],
Log@rng]; // RepeatedTiming
a == b == c == d == e == f


{0.768518, Null}

{0.895237, Null}

{0.10716, Null}

{0.131237, Null}

{0.000220158, Null}

{0.000120771, Null}

True

• This is a great trick and incredibly fast! But rng has to be strictly positive numbers. I have tried this out and discovered my real data has some rare negative numbers -_-" Commented May 24, 2023 at 9:10
• To be clear, this is a problem with my data, not your method :) Commented May 24, 2023 at 9:30
• @IntroductionToProbability This also works for negative data.
– Karl
Commented May 24, 2023 at 20:25
• The imaginary units still stick out; compare: Exp@ListConvolve[ConstantArray[1., 2], Log@{-1, 2, -3, 4}] and Times @@@ Partition[{-1, 2, -3, 4}, 2, 1]` Commented May 25, 2023 at 9:36
• @IntroductionToProbability try: Chop@Exp@ListConvolve[ConstantArray[1., 2], Log@{-1, 2, -3, 4}]
– Karl
Commented May 25, 2023 at 9:40