2
$\begingroup$

Building up on the solution proposed here : Simplifying nested If statements

You can find here the data set : allGazes.dat

allGazesX = 
 Uncompress@
   Import[FileNameJoin[{NotebookDirectory[], "allGazes.dat.gz"}], 
    "String"];

I need to filter large data set and believe I lack an efficient method to do so. The purpose here is to filter given the EuclideanDistance[] between gazes. Below is what I am using currently :

This is what i am using currently :

GZ[delta_] := ParallelTable[
              Table[
                    Reap[z = allGazesX[[subNO, dispNo, 1, ;; 2]];Sow[z];
                    Scan[
                         If[
                            EuclideanDistance[#, z] > delta,
                            z = #;Sow[z]] &,
                            allGazesX[[subNO, dispNo, All, ;; 2]]]][[2, 1]],

              {dispNo, Range[Length[allGazesX[[subNO]]]]}],
              {subNO, Range[5]}];
$\endgroup$
11
  • $\begingroup$ Might I ask why you aren't using ParallelTable for both dispNo and subNO? $\endgroup$
    – rcollyer
    Commented Mar 2, 2012 at 15:13
  • $\begingroup$ Not having actual data, I'm not exactly sure what this is doing. But it looks like it might possibly benefit from use of single-argument Nearest[]. $\endgroup$ Commented Mar 2, 2012 at 16:32
  • $\begingroup$ @rcollyer, If you do, I believe you get an error message saying you can`t have nested ParallelTable[] $\endgroup$
    – 500
    Commented Mar 2, 2012 at 16:39
  • $\begingroup$ Both Table and ParallelTable accept multiple iterator arguments, or should. So, you could write ParallelTable[..., {subNO, ...}, {dispNo, ...}] instead of nesting the second Table inside. Note, dispNo has to go after subNO which it depends on. Does that clarify what I was asking? Or, do you still get an error message? $\endgroup$
    – rcollyer
    Commented Mar 2, 2012 at 16:53
  • $\begingroup$ @rcollyer I think, using Table inside ParallelTable may make sense, if you want to force certain (coarse) granularity of your computations. $\endgroup$ Commented Mar 2, 2012 at 17:57

1 Answer 1

3
$\begingroup$

EDIT

Apparently, I have misunderstood the problem. Here is the solution which, for smaller tests, produces the results identical to the original one:

getDistantPoints = 
  Compile[{{pts, _Real, 2}, {delta, _Real}},
     Module[{res = Table[{0., 0.}, {Length[pts]}], ctr = 1},
        res[[1]] = pts[[1]];
        Do[
          If[Norm[pts[[i]] - res[[ctr]]] > delta, 
            res[[++ctr]] = pts[[i]]
          ], 
          {i, Length[pts]}];
        Take[res, ctr]],
     CompilationTarget -> "C", RuntimeOptions -> "Speed"]


Clear[GZFastAlt];
GZFastAlt[delta_, data_] :=
  Module[{ldata = data},
     ParallelTable[
       Table[
          getDistantPoints[ldata [[subNO, dispNo, All, ;; 2]], delta],
          {dispNo, Range[Length[ldata [[subNO]]]]}
       ], {subNO, Range[5]}]];

and runs in about 2 seconds on my 6 cores:

(res = GZFastAlt[0.1,allGazesX]);//AbsoluteTiming
{2.2451172,Null}

END EDIT

As a bonus, this keeps things packed, which is a big deal for your data - even in packed form, the computation consumes quite a bit of memory.

$\endgroup$
5
  • 1
    $\begingroup$ Are these identical? In the original, the value of z is a moving target, so to speak. $\endgroup$ Commented Mar 2, 2012 at 16:32
  • $\begingroup$ @Daniel Good point! I thought z was only used as a recording device, and did not think that it can be important. Actually, z being a moving target makes more sense. Will think of a modification for this case, and hopefully updates soon. $\endgroup$ Commented Mar 2, 2012 at 17:05
  • $\begingroup$ @Daniel Ok, fixed (hopefully). Thanks for spotting it! $\endgroup$ Commented Mar 2, 2012 at 17:36
  • $\begingroup$ You're welcome. As regards things that get spotted, I'm just glad it was a wandering 'z and not a hungry leopard. [Exit stage left, ducking rotting vegetables from audience.] $\endgroup$ Commented Mar 2, 2012 at 18:46
  • $\begingroup$ @Daniel That assumes the audience was well-prepared, and actually expecting the failure :) $\endgroup$ Commented Mar 2, 2012 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.