1
$\begingroup$

I want to determine whether I have regularly sampled data or not, a là RegularlySampledQ. The data itself might be huge. I tried a bunch of different approaches and the best I could come up with was this:

regSampQ[d_, window_: 100] :=
  If[Length@d > window,
   Module[{eq = True},
    Do[
     If[! Equal @@ Differences[d[[n - window ;; n]]], 
       eq = False; Break[]
      ], {n, window + 1, Length@d, window}];
    eq
    ],
   Equal @@ Differences[d]
   ];
regSampQ~SetAttributes~HoldFirst

It turns out this is a somewhat efficient way to do this, beating out my other best attempt, which was just

regSampNaive = Function[Null, Equal @@ Differences[#], HoldFirst];

The problem is that the data can be incredibly large. Here're some test cases I had:

test1 = RandomReal[{}, 50000];

regSampNaive@test1 // RepeatedTiming

{0.0037, False}

regSampQ@test1 // RepeatedTiming

{0.000097, False}

test2 = ConstantArray[0, 50000];

regSampNaive@test2 // RepeatedTiming

{0.0025, True}

regSampQ@test2 // RepeatedTiming

{0.0026, True}

test3 = Join[ConstantArray[0, 25000], ConstantArray[1, 25000]];

regSampNaive@test3 // RepeatedTiming

{0.0023, False}

regSampQ@test3 // RepeatedTiming

{0.0014, False}

But I feel like there has to be a really clean, fast way to do this... am I wrong?

$\endgroup$
7
  • 1
    $\begingroup$ On my machine (OS X V11.3) Length@DeleteDuplicates@Differences[d[[n - window ;; n]]] != 1 cuts the timings in half for True $\endgroup$ Oct 18, 2018 at 8:10
  • $\begingroup$ @MikeHoneychurch oh wow you're right. That's...embarrassing for Equal. But a very nice optimization. It cuts all my timings by a small factor at least. Actually, thinking about it the overhead might be in Apply instead. It might just be a data-copying issue. $\endgroup$
    – b3m2a1
    Oct 18, 2018 at 8:14
  • $\begingroup$ See if your method is faster if you widen the window to e.g. 300. I don't recall exactly whether Apply gets autocompiled but if it does it would be for 200-250+ list length. ps. DeleteDuplicates was a first "easy" thing that came to mind but there are probably other ways if you got creative $\endgroup$ Oct 18, 2018 at 23:13
  • $\begingroup$ @MikeHoneychurch compilation didn’t actually help when I used it procedurally. The data transfer step is the killer. $\endgroup$
    – b3m2a1
    Oct 18, 2018 at 23:35
  • $\begingroup$ I'm talking about autocompilation. mathematica.stackexchange.com/questions/311/… $\endgroup$ Oct 18, 2018 at 23:47

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.