Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Building up on the solution proposed here : Simplifying nested If statements

You can find here the data set : allGazes.dat

allGazesX = 
 Uncompress@
   Import[FileNameJoin[{NotebookDirectory[], "allGazes.dat.gz"}], 
    "String"];

I need to filter large data set and believe I lack an efficient method to do so. The purpose here is to filter given the EuclideanDistance[] between gazes. Below is what I am using currently :

This is what i am using currently :

GZ[delta_] := ParallelTable[
              Table[
                    Reap[z = allGazesX[[subNO, dispNo, 1, ;; 2]];Sow[z];
                    Scan[
                         If[
                            EuclideanDistance[#, z] > delta,
                            z = #;Sow[z]] &,
                            allGazesX[[subNO, dispNo, All, ;; 2]]]][[2, 1]],

              {dispNo, Range[Length[allGazesX[[subNO]]]]}],
              {subNO, Range[5]}];
share|improve this question
    
Might I ask why you aren't using ParallelTable for both dispNo and subNO? –  rcollyer Mar 2 '12 at 15:13
    
Not having actual data, I'm not exactly sure what this is doing. But it looks like it might possibly benefit from use of single-argument Nearest[]. –  Daniel Lichtblau Mar 2 '12 at 16:32
    
@rcollyer, If you do, I believe you get an error message saying you can`t have nested ParallelTable[] –  500 Mar 2 '12 at 16:39
    
Both Table and ParallelTable accept multiple iterator arguments, or should. So, you could write ParallelTable[..., {subNO, ...}, {dispNo, ...}] instead of nesting the second Table inside. Note, dispNo has to go after subNO which it depends on. Does that clarify what I was asking? Or, do you still get an error message? –  rcollyer Mar 2 '12 at 16:53
    
@rcollyer I think, using Table inside ParallelTable may make sense, if you want to force certain (coarse) granularity of your computations. –  Leonid Shifrin Mar 2 '12 at 17:57

1 Answer 1

up vote 3 down vote accepted

EDIT

Apparently, I have misunderstood the problem. Here is the solution which, for smaller tests, produces the results identical to the original one:

getDistantPoints = 
  Compile[{{pts, _Real, 2}, {delta, _Real}},
     Module[{res = Table[{0., 0.}, {Length[pts]}], ctr = 1},
        res[[1]] = pts[[1]];
        Do[
          If[Norm[pts[[i]] - res[[ctr]]] > delta, 
            res[[++ctr]] = pts[[i]]
          ], 
          {i, Length[pts]}];
        Take[res, ctr]],
     CompilationTarget -> "C", RuntimeOptions -> "Speed"]


Clear[GZFastAlt];
GZFastAlt[delta_, data_] :=
  Module[{ldata = data},
     ParallelTable[
       Table[
          getDistantPoints[ldata [[subNO, dispNo, All, ;; 2]], delta],
          {dispNo, Range[Length[ldata [[subNO]]]]}
       ], {subNO, Range[5]}]];

and runs in about 2 seconds on my 6 cores:

(res = GZFastAlt[0.1,allGazesX]);//AbsoluteTiming
{2.2451172,Null}

END EDIT

As a bonus, this keeps things packed, which is a big deal for your data - even in packed form, the computation consumes quite a bit of memory.

share|improve this answer
1  
Are these identical? In the original, the value of z is a moving target, so to speak. –  Daniel Lichtblau Mar 2 '12 at 16:32
    
@Daniel Good point! I thought z was only used as a recording device, and did not think that it can be important. Actually, z being a moving target makes more sense. Will think of a modification for this case, and hopefully updates soon. –  Leonid Shifrin Mar 2 '12 at 17:05
    
@Daniel Ok, fixed (hopefully). Thanks for spotting it! –  Leonid Shifrin Mar 2 '12 at 17:36
    
You're welcome. As regards things that get spotted, I'm just glad it was a wandering 'z and not a hungry leopard. [Exit stage left, ducking rotting vegetables from audience.] –  Daniel Lichtblau Mar 2 '12 at 18:46
    
@Daniel That assumes the audience was well-prepared, and actually expecting the failure :) –  Leonid Shifrin Mar 2 '12 at 19:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.