# Selecting elements with a particular slope

I have a dataset for which I want to identify 'groupings' based on three criteria, but only one of them is giving me a headache. The criteria are that each group must:

• have a minimum number of points to be considered a trace
• the distance between any two adjacent points must be under a threshold
• the slope between any consecutive elements within a grouping must be slope +- tolerance.

The first two work as I would like them to, but the slope criterion only works if there is a strict equality with the slope. The checkSlope[pointA_, pointB_, slope_, tol_] function seems to pick the wrong elements when I use the inequality of slope -tol < x < slope + tol, but works fine if I simply ask x == slope.

Any help would be appreciated.

rdata = Import["the pastebin link from above"]; (*raw data*)

uniqueFreqs =
Sort@Union@
rdata[[;; ,
1]]; (* find the unique frequency points in the 2D-thresholded data *)

numberOfPoints = 5; (* minimum number of points for a trace*)
vecDistance = 15; (* maximum distance between any two points within a trace*)
slopeLeeway =
1. (100 + {-1, 1} 5)/
100; (* creates a percentage deviation. For example if \
slopeLeeway is 5%, then it generates {0.95,1.05} *)
slope = 2; (* look for elements with this slope value*)

checkSlope[pointA_, pointB_, slope_, tol_] :=
If[pointB != pointA,
If[pointB[[1]] != pointA[[1]],
tol[[If[slope > 0, 1, 2]]] slope <= (pointB[[2]] - pointA[[2]])/(
pointB[[1]] - pointA[[1]])(*==slope*)<=
tol[[If[slope > 0, 2, 1]]] slope, False], False];(* checks if the slope between two points is within tolerance of the slope *)

vecSeparation[pointA_, pointB_] :=
Sqrt[1. Total@((pointB - pointA)^2)];(* calculates the distance between any two points*)

newData =
DeleteCases[
Select[(Gather[rdata, checkSlope[#2, #1, slope, slopeLeeway] &]),
Length@# >= numberOfPoints &], {}]; (* create groupings based on slope*)
vecCheck =
Table[vecSeparation @@ # <=
Abs[vecDistance If[slope == 0, 1, slope]] & /@
Partition[data, 2, 1], {data,
newData}](* generates a True/False map which evaluates whether \
two successive points satisfy the distance criterion *);
splitting =
Table[(#[[;; , 1]]~Join~{Max@#[[;; , 1]] + 1} & /@
Select[(SplitBy[{Range[Length@data], data}\[Transpose], #[[2]] ==
True &]), AllTrue[#[[;; , 2]], TrueQ] &]
), {data,
vecCheck}] (* converts the T/F list into positions and splits \
them based on whether they were satisfying the distance criterion *);
splitData =
Flatten[Table[
newData[[i]][[#]] & /@
Select[splitting[[i]], Length@# >= numberOfPoints &], {i, 1,
Length@newData, 1}],
1] (* removes all sets that do not have at least the minimum \
number of points, and then picks the corresponding elements from the \
identified traces. *);
Length /@ {rdata, newData, vecCheck, splitting, splitData}
Manipulate[
ListPlot[{rdata, newData[[i]], Splice@splitData},
ImageSize -> 550], {i, 1, Length@newData, 1}, Paneled -> False]


In the above example I have asked it to find elements with a slope of +2. The tolerance is set to 5%, so it should only be looking for slopes between 1.9 and 2.1 The fact that it also picks elements with slope of 1 is weird to me.

I suspect that perhaps Gather does not work well with inequalities?

• This question is somehow similar. I also provided an answer which sets the next point given the current "derivative". Commented May 11, 2023 at 11:55
• Hi @Domen, I just tested your algorithm on my dataset I must admit that your algorithm is great! However the fact that the results are a bit random can be difficult to incorporate in a pipeline where you want consistent results. If I wanted to go back and double check that particular dataset, chances are I would get different results, which is not ideal. I'll have to look into how it works to see if I can adjust it somehow. Is there an obvious for you means to improve upon your function?
– alex
Commented May 11, 2023 at 12:57