I have a dataset for which I want to identify 'groupings' based on three criteria, but only one of them is giving me a headache. The criteria are that each group must:

  • have a minimum number of points to be considered a trace
  • the distance between any two adjacent points must be under a threshold
  • the slope between any consecutive elements within a grouping must be slope +- tolerance.

The first two work as I would like them to, but the slope criterion only works if there is a strict equality with the slope. The checkSlope[pointA_, pointB_, slope_, tol_] function seems to pick the wrong elements when I use the inequality of slope -tol < x < slope + tol, but works fine if I simply ask x == slope.

Any help would be appreciated.

rdata = Import["the pastebin link from above"]; (*raw data*)

uniqueFreqs = 
    rdata[[;; , 
      1]]; (* find the unique frequency points in the 2D-thresholded data *) 

numberOfPoints = 5; (* minimum number of points for a trace*)
vecDistance = 15; (* maximum distance between any two points within a trace*)
slopeLeeway = 
  1. (100 + {-1, 1} 5)/
    100; (* creates a percentage deviation. For example if \
slopeLeeway is 5%, then it generates {0.95,1.05} *)
slope = 2; (* look for elements with this slope value*)

checkSlope[pointA_, pointB_, slope_, tol_] := 
  If[pointB != pointA, 
   If[pointB[[1]] != pointA[[1]], 
    tol[[If[slope > 0, 1, 2]]] slope <= (pointB[[2]] - pointA[[2]])/(
     pointB[[1]] - pointA[[1]])(*==slope*)<= 
     tol[[If[slope > 0, 2, 1]]] slope, False], False];(* checks if the slope between two points is within tolerance of the slope *)

vecSeparation[pointA_, pointB_] := 
  Sqrt[1. Total@((pointB - pointA)^2)];(* calculates the distance between any two points*)

newData = 
   Select[(Gather[rdata, checkSlope[#2, #1, slope, slopeLeeway] &]), 
    Length@# >= numberOfPoints &], {}]; (* create groupings based on slope*)
vecCheck = 
  Table[vecSeparation @@ # <= 
      Abs[vecDistance If[slope == 0, 1, slope]] & /@ 
    Partition[data, 2, 1], {data, 
    newData}](* generates a True/False map which evaluates whether \
two successive points satisfy the distance criterion *);
splitting = 
  Table[(#[[;; , 1]]~Join~{Max@#[[;; , 1]] + 1} & /@ 
     Select[(SplitBy[{Range[Length@data], data}\[Transpose], #[[2]] ==
           True &]), AllTrue[#[[;; , 2]], TrueQ] &]
    ), {data, 
    vecCheck}] (* converts the T/F list into positions and splits \
them based on whether they were satisfying the distance criterion *);
splitData = 
    newData[[i]][[#]] & /@ 
     Select[splitting[[i]], Length@# >= numberOfPoints &], {i, 1, 
     Length@newData, 1}], 
   1] (* removes all sets that do not have at least the minimum \
number of points, and then picks the corresponding elements from the \
identified traces. *);
Length /@ {rdata, newData, vecCheck, splitting, splitData}
 ListPlot[{rdata, newData[[i]], Splice@splitData}, 
  ImageSize -> 550], {i, 1, Length@newData, 1}, Paneled -> False]

enter image description here

In the above example I have asked it to find elements with a slope of +2. The tolerance is set to 5%, so it should only be looking for slopes between 1.9 and 2.1 The fact that it also picks elements with slope of 1 is weird to me.

I suspect that perhaps Gather does not work well with inequalities?

  • 1
    $\begingroup$ This question is somehow similar. I also provided an answer which sets the next point given the current "derivative". $\endgroup$
    – Domen
    May 11, 2023 at 11:55
  • $\begingroup$ Hi @Domen, I just tested your algorithm on my dataset I must admit that your algorithm is great! However the fact that the results are a bit random can be difficult to incorporate in a pipeline where you want consistent results. If I wanted to go back and double check that particular dataset, chances are I would get different results, which is not ideal. I'll have to look into how it works to see if I can adjust it somehow. Is there an obvious for you means to improve upon your function? $\endgroup$
    – alex
    May 11, 2023 at 12:57


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.