4
$\begingroup$

I have some data representing a vaguely periodic signal, here is a sample

signal = Flatten[
   Import["https://gist.githubusercontent.com/tomginsberg/3dd708afa51dc4300ad4a50b24a0b3bf/raw/8903014fc7a71bb430d3435e7fc3c5e36bec0281/signal.txt", "Data"]];

ListPlot[signal, AspectRatio -> 1/10, ImageSize -> Full, 
 Axes -> False]

enter image description here

My only knowledge about the data us that it should contain several increasing lines of similar slope and several areas of noise. I went through the first bit by hand and selected windows where lines occur then ran a linear regression to get this

(* Indexed signal *)
isig = Transpose[{Range[Length[signal]], signal}];
f[{a_, b_}] := isig[[a ;; b]]

(* Fits in pre determined windows *)
fits = (Fit[#1, {1, x}, x] &) /@ 
   f /@ Partition[{1, 194, 1779, 1846, 1935, 2182, 2178, 2425, 2490, 
      2600}, 2];

Show[ListPlot[signal[[;; 2600]], AspectRatio -> 1/10, 
  ImageSize -> Full], 
 Plot[fits, {x, 0, 3000}, PlotRange -> {0, 248}, PlotStyle -> {Thick},
   PlotTheme -> "Monochrome"]]

enter image description here

What I want is some automated way to come up with the windows where all the points in the window approximately follow a line. Then I can do regression on the windows and the problem is solved.

Any thoughts or suggestions would be appreaciated. Thanks.

$\endgroup$
  • $\begingroup$ Maybe you can use Differences@signal to identify large changes (e.g. > +/- 200) in successive values to locate the windows? $\endgroup$ – Rohit Namjoshi Jul 10 at 22:13
3
$\begingroup$

My idea here is to use a GradientFilter on the accumulated signal and spot the maxima using MaxDetect. This creates a run of 1's when the signal gains momentum in a straight line towards the peaks, and we can find positions of those 1's and split when there's a jump in the position > 1.

acmg = Rescale[GradientFilter[Accumulate@signal, 1]];
mdacmg = MaxDetect[acmg, .85];
ListLinePlot[{mdacmg, acmg}, Filling -> {2 -> {1}}]
windows = #[[{1, -1}]] & /@ Split[Flatten[Position[mdacmg, 1]], #2 - #1 == 1 &];

Show[
 With[{max = Max[signal]},
  Graphics[{Gray, Rectangle[{#[[1]], 0}, {#[[2]], max}] & /@ windows}]
  ],
 ListLinePlot[signal, PlotStyle -> Red]
]

(* resulting windows to check for fit:
 {{8, 185}, {351, 361}, {1128, 1165}, {1265, 1690}, {1729, 1941},
 {1975, 2181}, {2216, 2441}, {2483, 2639}, {2764, 2917}, {2982, 3630},
 {3803, 3935}, {4123, 4304}, {4367, 5585},
 {5616, 5870}, {5897, 6164}, {6193, 6404}} *)

runs

| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks that work quite well, I added an extra flag that only excepts windows if the $r^2$ value of the fit is high enough (say > .9) $\endgroup$ – user2757771 Jul 10 at 23:42
  • $\begingroup$ is GradientFilter[Accumulate@signal, 1] not essentially the identity? $\endgroup$ – user2757771 Jul 14 at 0:03
  • $\begingroup$ @user2757771 no - that would be Differences[Accumulate[signal]]. The GradientFilter with radius 1 is wider and more smooth - the result of GradientFilter[Accumulate@signal, 1] is very different from the original signal. $\endgroup$ – flinty Jul 14 at 11:22
  • $\begingroup$ I see GradientFilter[Accumulate[{a, b, c, d, e, f}], 1] // Simplify returns {Abs[b]/2, 1/2 Abs[b + c], 1/2 Abs[c + d], 1/2 Abs[d + e], 1/2 Abs[e + f], Abs[f]/2}. So a slightly smoothed version. The output is still nearly identical if you run ListLinePlot[{Rescale@signal, acmg}] to compare. $\endgroup$ – user2757771 Jul 14 at 16:46
  • $\begingroup$ If the smoothing bothers you, just remove it, or use a mean filter instead. $\endgroup$ – flinty Jul 14 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.