7
$\begingroup$

So I have this list of raw temperature data. I want to find the best fit for this trend but first I want to cure the data, because at several positions there are 'dips' (you can see them in the plot below) indicating the distance where the temperatures are measured. The goal is to remove the dips to find a better fit for the actual trend.

{77.9, 77.3, 76.8, 76.2, 75.8, 75.3, 75., 74.5, 74.2, 73.9, 73.5, \
73.2, 72.9, 72.6, 72.1, 71.8, 71.3, 71., 70.4, 69.9, 69.5, 68.9, \
68.1, 67.3, 66.5, 65.7, 64.7, 63.4, 62.2, 60.8, 59.7, 58.8, 58., \
57.4, 56.9, 56.4, 55.4, 54.8, 52.6, 49.7, 46.9, 45.4, 45.5, 47.7, \
51.2, 53.4, 53.8, 53.7, 53.5, 53.4, 53.2, 53., 52.8, 52.6, 52.6, \
52.4, 52.2, 52.1, 51.9, 51.7, 51.5, 51.2, 51., 50.8, 50.6, 50.5, \
50.5, 50.4, 50.3, 50.2, 50.2, 50.3, 50.3, 50.2, 50.2, 50.1, 50., \
49.9, 49.9, 49.8, 49.7, 49.6, 49.6, 49.5, 49.5, 48.9, 48.8, 48.7, \
48.6, 48.5, 48.4, 48.3, 48.2, 47.9, 47.1, 44.9, 40.9, 36.7, 34.6, \
35.5, 39.1, 43.8, 46.9, 47.3, 47.3, 47.3, 47.3, 47.1, 46.9, 46.5, \
46.4, 46.3, 46.2, 46.1, 45.9, 45.8, 45.6, 45.5, 45.4, 45.2, 45.2, \
44.9, 44.8, 44.6, 44.4, 44.1, 44.1, 44., 43.9, 43.9, 43.8, 43.7, \
43.6, 43.5, 43.5, 43.5, 43.5, 43.4, 43.4, 43.3, 43.1, 43., 42.9, \
42.9, 42.8, 42.7, 42.5, 42.4, 42.5, 42.4, 42.4, 42.2, 42.2, 42.1, \
41.6, 39.3, 35., 33., 32.9, 32.3, 34.5, 39.3, 41.2, 41.4, 41.4, 41.4, \
41.4, 41.4, 41.4, 41.4, 41.4, 41.3, 41.3, 41.2, 41.2, 41.1, 41., \
40.9, 40.7, 40.8, 40.9, 40.8, 40.8, 40.7, 40.5, 40.4, 40.3, 40.3, \
40.1, 40.1, 40., 40., 39.9, 39.9, 39.8, 39.8, 39.7, 39.6, 39.6, 39.5, \
39.4, 39.4, 39.3, 39.4, 39.3, 39.2, 39.1, 38.9, 38.8, 38.7, 38.6, \
38.5, 38.5, 38.4, 38.3, 38.1, 37.6, 35.9, 33.6, 32.1, 31.6, 32., \
33.8, 36.1, 37.3, 37.6, 37.6, 37.5, 37.4, 37.4, 37.5, 37.5, 37.5, \
37.4, 37.4, 37.3, 37.2, 37.2, 37.1, 37., 36.9, 36.9, 36.8, 36.8, \
36.8, 36.7, 36.6, 36.5, 36.3, 36.2, 36.3, 36.3, 36.1, 36.1, 36.3, \
36.3, 36.2, 36.3, 36.2, 36.1, 36.1, 36.1, 36., 35.8, 35.7, 35.7, \
35.6, 35.5, 35.5, 35.5, 35.4, 35.4, 35.3, 35.3, 35.2, 35.2, 35.1, \
35.1, 35.1, 35.1, 35., 34.5, 33.1, 31.8, 30.8, 30.2, 30.3, 31.3, \
32.6, 33.5, 34.1, 34.4, 34.3, 34.3, 34.2, 34.3, 34.2, 34.2, 34.1, \
34.1, 34., 33.9, 34.4, 34.3, 34.3, 34.3, 34.2, 34.1, 34.1, 34.1, \
34.1, 34., 33.8, 33.9, 33.7}

Display of the temperature graph and the dips to indicate the distance at which the temperature is measured (the list I added is the yellow one in the graph)

My problem is that the dips dont occure at the same positions in the temperature list (like every 50th to 60th elements are the dip-temperatures). It seems to me that with Drop or Delete I cannot do it all at once (like: Drop[list, {50;;60},{110;;130}] ).

Is there a way to automate this so I do not have to do it manually? Thanks in advance!

$\endgroup$
  • $\begingroup$ Wouldn’t fitting with a relatively smooth model naturally disregard those dips? Have you tried fitting the data as is? $\endgroup$ – MarcoB Jul 13 at 15:17
  • 1
    $\begingroup$ Yes I already fitted my raw data as it is. Since I am only starting with mathematica I wanted to cure the data for learning purposes but also to see how the absence of the dips affects the result. $\endgroup$ – Maxi Jul 13 at 15:39
7
$\begingroup$

A MaxFilter does a pretty good job as your data is mostly decreasing:

MaxFilter[data, 5] // ListPlot

However this slightly changes the phase of the curve by 5 points. If you use a MovingMap instead, the right aligned window will avoid this problem, but it will sacrifice five points at the start of your data if you can tolerate that:

Show[
 ListPlot[MovingMap[Max, data, 10]],
 ListPlot[data, PlotStyle -> {PointSize[Small], Red}]
 ]

max moving map plot

| improve this answer | |
$\endgroup$
5
$\begingroup$

You can use this method here. Play with parameter xScale for adjustment.

 xScale = 30.;
    xy = Transpose[{N[Range[Length[data]]]*(xScale/Length[data]), data}];
    start = {-xScale, First[data]};
    finish = {2*xScale, Last[data]};
    graph = NearestNeighborGraph[Join[{start}, xy, {finish}], 25];
    graph = SetProperty[graph, 
       EdgeWeight -> 
        Apply[SquaredEuclideanDistance, EdgeList[graph], {1}]];
    path = FindShortestPath[graph, start, finish][[2 ;; -2]];
    ListLinePlot[path, PlotRange -> MinMax[data], 
     Prolog -> {Gray, Point[xy]}, PlotStyle -> Red, ImageSize -> 600]

enter image description here

| improve this answer | |
$\endgroup$
4
$\begingroup$

An approach quite similar to @flinty's answer is to use the image processing filter "Dilation":

ListPlot[{data, Dilation[data, 5][[5 ;; All]]}]

enter image description here

The Dilation rides over the top of the data and the indexing ([[5 ;; All]]) removes the shift of the data.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.