Building up on the solution proposed here : Simplifying nested If statements

You can find here the data set : allGazes.dat

allGazesX = 
   Import[FileNameJoin[{NotebookDirectory[], "allGazes.dat.gz"}], 

I need to filter large data set and believe I lack an efficient method to do so. The purpose here is to filter given the EuclideanDistance[] between gazes. Below is what I am using currently :

This is what i am using currently :

GZ[delta_] := ParallelTable[
                    Reap[z = allGazesX[[subNO, dispNo, 1, ;; 2]];Sow[z];
                            EuclideanDistance[#, z] > delta,
                            z = #;Sow[z]] &,
                            allGazesX[[subNO, dispNo, All, ;; 2]]]][[2, 1]],

              {dispNo, Range[Length[allGazesX[[subNO]]]]}],
              {subNO, Range[5]}];
  • $\begingroup$ Might I ask why you aren't using ParallelTable for both dispNo and subNO? $\endgroup$
    – rcollyer
    Mar 2 '12 at 15:13
  • $\begingroup$ Not having actual data, I'm not exactly sure what this is doing. But it looks like it might possibly benefit from use of single-argument Nearest[]. $\endgroup$ Mar 2 '12 at 16:32
  • $\begingroup$ @rcollyer, If you do, I believe you get an error message saying you can`t have nested ParallelTable[] $\endgroup$
    – 500
    Mar 2 '12 at 16:39
  • $\begingroup$ Both Table and ParallelTable accept multiple iterator arguments, or should. So, you could write ParallelTable[..., {subNO, ...}, {dispNo, ...}] instead of nesting the second Table inside. Note, dispNo has to go after subNO which it depends on. Does that clarify what I was asking? Or, do you still get an error message? $\endgroup$
    – rcollyer
    Mar 2 '12 at 16:53
  • $\begingroup$ @rcollyer I think, using Table inside ParallelTable may make sense, if you want to force certain (coarse) granularity of your computations. $\endgroup$ Mar 2 '12 at 17:57


Apparently, I have misunderstood the problem. Here is the solution which, for smaller tests, produces the results identical to the original one:

getDistantPoints = 
  Compile[{{pts, _Real, 2}, {delta, _Real}},
     Module[{res = Table[{0., 0.}, {Length[pts]}], ctr = 1},
        res[[1]] = pts[[1]];
          If[Norm[pts[[i]] - res[[ctr]]] > delta, 
            res[[++ctr]] = pts[[i]]
          {i, Length[pts]}];
        Take[res, ctr]],
     CompilationTarget -> "C", RuntimeOptions -> "Speed"]

GZFastAlt[delta_, data_] :=
  Module[{ldata = data},
          getDistantPoints[ldata [[subNO, dispNo, All, ;; 2]], delta],
          {dispNo, Range[Length[ldata [[subNO]]]]}
       ], {subNO, Range[5]}]];

and runs in about 2 seconds on my 6 cores:

(res = GZFastAlt[0.1,allGazesX]);//AbsoluteTiming


As a bonus, this keeps things packed, which is a big deal for your data - even in packed form, the computation consumes quite a bit of memory.

  • 1
    $\begingroup$ Are these identical? In the original, the value of z is a moving target, so to speak. $\endgroup$ Mar 2 '12 at 16:32
  • $\begingroup$ @Daniel Good point! I thought z was only used as a recording device, and did not think that it can be important. Actually, z being a moving target makes more sense. Will think of a modification for this case, and hopefully updates soon. $\endgroup$ Mar 2 '12 at 17:05
  • $\begingroup$ @Daniel Ok, fixed (hopefully). Thanks for spotting it! $\endgroup$ Mar 2 '12 at 17:36
  • $\begingroup$ You're welcome. As regards things that get spotted, I'm just glad it was a wandering 'z and not a hungry leopard. [Exit stage left, ducking rotting vegetables from audience.] $\endgroup$ Mar 2 '12 at 18:46
  • $\begingroup$ @Daniel That assumes the audience was well-prepared, and actually expecting the failure :) $\endgroup$ Mar 2 '12 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.