# Detecting Lines in Data

I have some data representing a vaguely periodic signal, here is a sample

signal = Flatten[

ListPlot[signal, AspectRatio -> 1/10, ImageSize -> Full,
Axes -> False]


My only knowledge about the data us that it should contain several increasing lines of similar slope and several areas of noise. I went through the first bit by hand and selected windows where lines occur then ran a linear regression to get this

(* Indexed signal *)
isig = Transpose[{Range[Length[signal]], signal}];
f[{a_, b_}] := isig[[a ;; b]]

(* Fits in pre determined windows *)
fits = (Fit[#1, {1, x}, x] &) /@
f /@ Partition[{1, 194, 1779, 1846, 1935, 2182, 2178, 2425, 2490,
2600}, 2];

Show[ListPlot[signal[[;; 2600]], AspectRatio -> 1/10,
ImageSize -> Full],
Plot[fits, {x, 0, 3000}, PlotRange -> {0, 248}, PlotStyle -> {Thick},
PlotTheme -> "Monochrome"]]


What I want is some automated way to come up with the windows where all the points in the window approximately follow a line. Then I can do regression on the windows and the problem is solved.

Any thoughts or suggestions would be appreaciated. Thanks.

• Maybe you can use Differences@signal to identify large changes (e.g. > +/- 200) in successive values to locate the windows? – Rohit Namjoshi Jul 10 at 22:13

My idea here is to use a GradientFilter on the accumulated signal and spot the maxima using MaxDetect. This creates a run of 1's when the signal gains momentum in a straight line towards the peaks, and we can find positions of those 1's and split when there's a jump in the position > 1.

acmg = Rescale[GradientFilter[Accumulate@signal, 1]];
mdacmg = MaxDetect[acmg, .85];
ListLinePlot[{mdacmg, acmg}, Filling -> {2 -> {1}}]
windows = #[[{1, -1}]] & /@ Split[Flatten[Position[mdacmg, 1]], #2 - #1 == 1 &];

Show[
With[{max = Max[signal]},
Graphics[{Gray, Rectangle[{#[[1]], 0}, {#[[2]], max}] & /@ windows}]
],
ListLinePlot[signal, PlotStyle -> Red]
]

(* resulting windows to check for fit:
{{8, 185}, {351, 361}, {1128, 1165}, {1265, 1690}, {1729, 1941},
{1975, 2181}, {2216, 2441}, {2483, 2639}, {2764, 2917}, {2982, 3630},
{3803, 3935}, {4123, 4304}, {4367, 5585},
{5616, 5870}, {5897, 6164}, {6193, 6404}} *)


• thanks that work quite well, I added an extra flag that only excepts windows if the $r^2$ value of the fit is high enough (say > .9) – user2757771 Jul 10 at 23:42
• is GradientFilter[Accumulate@signal, 1] not essentially the identity? – user2757771 Jul 14 at 0:03
• @user2757771 no - that would be Differences[Accumulate[signal]]. The GradientFilter with radius 1 is wider and more smooth - the result of GradientFilter[Accumulate@signal, 1] is very different from the original signal. – flinty Jul 14 at 11:22
• I see GradientFilter[Accumulate[{a, b, c, d, e, f}], 1] // Simplify returns {Abs[b]/2, 1/2 Abs[b + c], 1/2 Abs[c + d], 1/2 Abs[d + e], 1/2 Abs[e + f], Abs[f]/2}. So a slightly smoothed version. The output is still nearly identical if you run ListLinePlot[{Rescale@signal, acmg}] to compare. – user2757771 Jul 14 at 16:46
• If the smoothing bothers you, just remove it, or use a mean filter instead. – flinty Jul 14 at 17:28