2
$\begingroup$

Context: I have millisecond timestamped sample data that I want to convert to a uniformly sampled series. This I do by padding each interval with repeated values (pbp ~ 'pad by position'}.

The following function works as intended, (I would credit the trick of using a range to generate an explicit list of indices for speed if I could remember who it came from...)

SetAttributes[pbp, HoldAll];
pbp[s_,idx_]:=
    Module[{ca,len, idx2},
        idx2 = idx - idx[[1]] +1;
        ca = ConstantArray[0,len=Last[idx]-First[idx]+1];
        (ca[[#1[[1]]]]=#1[[2]]) & /@ Table[{Range[idx2[[i]],idx2[[i+1]]-1], s[[i]]}, {i,1,Length[idx2]-1}];
        ca[[len]]=Last[s];
        Return[ca];
    ];

For nSamples = 100k pbp runs in about 1.5s but my data set is orders of magnitude larger and I have other things to do afterwards so I would like any performance gain I can get, but I can't get it to compile. Being aware of some ConstantArray compilation issues I substituted a table, and as far as I can tell from CompilerFunctions[] there is nothing controversial in this:

pbpCompiled=
    Compile[{{s, _Real, 1}, {idx, _Integer, 1}},
        Module[{ca,len, idx2},
            idx2 = idx - idx[[1]] +1;
            len=Last[idx]-First[idx]+1;
            ca = Table[0, {j,1,len}];
            (Part[ca, Part[#, 1]] = Part[#, 2]) & /@ Table[{Range[idx2[[i]],idx2[[i+1]]-1], s[[i]]}, {i,1,Length[idx2]-1}];
            ca[[len]]=Last[s];
            ca
        ],
    CompilationTarget -> "C", "RuntimeOptions"->"Speed"];

(NB, it was more idiomatically written but became less so as I tried to tease elements apart for debugging.)

However, I get this error (3x before further cpintIt are suppressed)

Compile`GetElement[System`Private`CompileSymbol[0],System`Private`CompileSymbol[1]][[1]] at position 2 of ca[[Compile`GetElement[System`Private`CompileSymbol[0],System`Private`CompileSymbol[1]][[1]]]] should be either a nonzero integer or a vector of nonzero integers; evaluation will use the uncompiled function

Q1 - what is wrong with the definition of the function to be compiled/how can it be fixed?

Q2 - is there some other uncompiled approach to the problem that would be significantly faster than the uncompiled version above?

The following can be used to provide some test data...

nSamples = 100000;
sampleSeries = RandomReal[{0, 1}, nSamples];
sampleTimes = IntegerPart@(1000 Accumulate[RandomReal[{0.001, 0.500}, nSamples]]);

then called as e.g. pbp[sampleSeries, sampleTimes]

$\endgroup$
  • $\begingroup$ What are you going to do with the output of pbp? Why not just create a zeroth order InterpolatingFunction from the data? $\endgroup$ – Carl Woll Mar 27 at 19:40
  • $\begingroup$ @CarlWoll. I'm going to moving average it with a 1s window, then subtract the m.a. and do stats on residuals and more processing of the m.a. Didn't follow your suggestion but tried it anyway on trust with ifun = Interpolation[Transpose[{sampleSeries, sampleTimes}], InterpolationOrder -> 0]; Plot[ifun[x], {x, 614, 2383}] (small segment at the beginning of 100 samples spanning 493 to ). Got a flat line and error; seems ifun[x] for any x -> "Input value {nnn} lies outside the range of data in the interpolating function". Maybe this is a Q2 answer but I can't tell :( $\endgroup$ – Julian Moore Mar 27 at 20:05
  • 1
    $\begingroup$ p[s_, idx_] := Join @@ MapThread[ ConstantArray[#1, #2] &, {s, Append[Differences[idx], 1]}]; a bit more concise, s/b quicker... $\endgroup$ – ciao Mar 27 at 21:38
  • $\begingroup$ @ciao I'll have to mentally unpack and then try that that when I get home; thanks. $\endgroup$ – Julian Moore Mar 28 at 7:21
  • $\begingroup$ Have you tried using TimeSeries with its built-in TimeSeriesResample method? reference.wolfram.com/language/guide/TimeSeries.html $\endgroup$ – Roman Mar 28 at 11:02
5
$\begingroup$

An array in Compile must have elements of a consistent type. The Table[] in the OP has, on the face of it, a mixture of integers and reals; however, the integers are promoted to type Real. Consequently, their values are not suitable as arguments to Part. Here is one workaround to prove the point, but one might consider alternatives:

pbpCompiled = 
 Compile[{{s, _Real, 1}, {idx, _Integer, 1}}, 
  Module[{ca, len, idx2}, idx2 = idx - idx[[1]] + 1;
   len = Last[idx] - First[idx] + 1;
   ca = Table[0, {j, 1, len}];
   (Part[ca, Round@Part[#, 1]] = Part[#, 2]) & /@   (* Note the use of Round[] *)
    Table[{Range[idx2[[i]], idx2[[i + 1]] - 1], s[[i]]}, {i, 1, 
      Length[idx2] - 1}];
   ca[[len]] = Last[s];
   ca], CompilationTarget -> "C", "RuntimeOptions" -> "Speed"]

Update: There is a further problem that I overlooked. The entry in the Table command is not a tensor, bu has the form {_List, _Real}:

{Range[idx2[[i]], idx2[[i + 1]] - 1], s[[i]]}

Beyond that, ca is initialized to a tensor of integers and later assigned real values, so it needs to be initialized to an array of real zeros, 0. Also Part[ca, Range[..]] led to a MainEvaluate, so I made it an explicit loop.

pbpCompiled = Compile[{{s, _Real, 1}, {idx, _Integer, 1}},
  Module[{ca, len, idx2},
   idx2 = idx - idx[[1]] + 1;
   len = Last[idx] - First[idx] + 1;
   ca = Table[0., {j, 1, len}];
   Do[
    Do[Part[ca, j] = Compile`GetElement[s, i], {j, idx2[[i]], 
      idx2[[i + 1]] - 1}], {i, 1, Length[idx2] - 1}];
   ca[[len]] = Last[s];
   ca], CompilationTarget -> "C", "RuntimeOptions" -> "Speed"]

If we make a similar changes in the uncompiled pbp, then the arrays stay packed and pbp[] runs just a little slower than the compiled version.

pbp[s_, idx_] := Module[{ca, len, idx2},
   idx2 = idx - idx[[1]] + 1;
   ca = ConstantArray[0., len = Last[idx] - First[idx] + 1];
   Do[ca[[Range[idx2[[i]], idx2[[i + 1]] - 1]]] = s[[i]],
    {i, 1, Length[idx2] - 1}];
   ca[[len]] = Last[s];
   Return[ca];];
$\endgroup$
  • $\begingroup$ Thank you; I didn't realise the Ints in that Table would get promoted, because they were in a separate sub-list. Now it compiles, proving your point, but at runtime it complains {{Ints}, Real} (see [1] below) "should be a rank 1 tensor of machine sized numbers", which is at least different (why not caught @ compile?). [1] for a short run, {{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25, 26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48, 49,50,<<86>>},0.776588} $\endgroup$ – Julian Moore Mar 28 at 5:20
  • $\begingroup$ @JulianMoore It was late and I didn't have time to think past what was the compiler error. I've updated my answer with improvements to both pbp and pbpCompiled. $\endgroup$ – Michael E2 Mar 28 at 12:16
  • $\begingroup$ using nSamples = 1000000 you pbp was faster than pbpcompiled and took only 0.083s, which I find quite astonishing. How might I have "known" that Do was a better way to approach this; performance optimisation seems a very dark art. [NB I also spotted that the ca tensor was initialised to int 0 not real 0, but the real problem was elsewhere] Where can one learn about such things, and Compile`GetElement etc.? Or is it just time, effort and MMA SE Q&A? $\endgroup$ – Julian Moore Mar 28 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.