Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

How can I reduce a non-linear trend of some data series if there are parts of "no data", represented by the value 99999? For example:

data={{124.55,368.632}, {124.6,281.976}, {124.65,200.952}, {124.7,91.384}, {124.75,-60.424},
{124.8,-282.504}, {124.85,-541.448}, {124.9,-378.76}, {124.95,99999}, {125,99999},
{125.05,99999}, {125.1,99999}, {125.15,99999}, {125.2,99999}, {125.25,99999}, {125.3,99999},
{125.35,265.46}, {125.4,376.75}, {125.45,462.54},...,{178.95,284.46}}

Normally, I am successful in reducing the non-linear trend of the data via

lm = LinearModelFit[data, {a, a^2, a^3, a^4, a^5},a];
Do[aha = Last[Take[data, {m, m}]]; y = Last[aha]; 
  x = First[aha]; new[m] = y - lm[x], {m, idim}];

but have no idea how to do the trend-reduction between those data gaps (i.e., outside of the regions with data = 99999). The position of those regions is not constant. After the reduction is performed, the position of the "no data"-values should be the same as before. Due to answer 1, unfortunately there is no periodically or cyclically behaviour in the data at all. Here, for example, is another, but complete data set:

 {{100.00,       0.303999        },
   {100.05,       0.303999        },
   {100.09,       0.176000        },
   {100.15,       0.303999        },
   {100.20,       0.176000        },
   {100.25,       0.303999        },
   {100.30,       0.431999        },
   {100.34,       0.431999        },
   {100.40,       0.559999        },
   {100.45,       0.687999        },
   {100.50,       0.687999        },
   {100.55,       0.943999        },
   {100.59,        1.07200        },
   {100.65,        1.20000        },
   {100.70,        1.32800        },
   {100.75,        1.32800        },
   {100.80,        1.20000        },
   {100.84,        1.20000        },
   {100.90,        1.20000        },
   {100.95,        1.07200        },
   {101.00,        1.07200        },
   {101.05,        1.07200        },
   {101.09,        1.07200        },
   {101.15,        1.07200        },
   {101.20,        1.07200        },
   {101.25,        1.07200        },
   {101.30,       0.943999        },
   {101.34,       0.943999        },
   {101.40,       0.943999        },
   {101.45,       0.687999        },
   {101.50,       0.431999        },
   {101.55,       0.303999        },
   {101.59,       4.800009E-002   },
   {101.65,      -8.000003E-002   },
   {101.70,      -0.207999        },
   {101.75,      -0.464000        },
   {101.80,      -0.719999        },
   {101.84,      -0.847999        },
   {101.90,      -0.847999        },
   {101.95,      -0.975999        },
   {102.00,       -1.10400        },
   {102.05,       -1.23200        },
   {102.09,       -1.36000        },
   {102.15,       -1.36000        },
   {102.20,       -1.36000        },
   {102.25,       -1.36000        },
   {102.30,       -1.36000        },
   {102.34,       -1.23200        },
   {102.40,       -1.10400        },
   {102.45,       -1.10400        },
   {102.50,      -0.975999        },
   {102.55,      -0.847999        },
   {102.59,      -0.719999        },
   {102.65,      -0.591999        },
   {102.70,      -0.464000        },
   {102.75,      -0.207999        },
   {102.80,       4.800009E-002   },
   {102.84,       4.800009E-002   },
   {102.90,       0.431999        },
   {102.95,       0.687999        },
   {103.00,       0.943999        },
   {103.05,        1.20000        },
   {103.09,        1.32800        },
   {103.15,        1.32800        },
   {103.20,        1.58400        },
   {103.25,        1.71200        },
   {103.30,        1.84000        },
   {103.34,        1.96800        },
   {103.40,        2.09600        },
   {103.45,        2.22400        },
   {103.50,        2.22400        },
   {103.55,        2.35199        },
   {103.59,        2.35199        },
   {103.65,        2.48000        },
   {103.70,        2.48000        },
   {103.75,        2.60800        },
   {103.80,        2.86399        },
   {103.84,        2.86399        },
   {103.90,        2.86399        },
   {103.95,        2.86399        },
   {104.00,        2.99200        },
   {104.05,        2.86399        },
   {104.09,        2.73600        },
   {104.15,        2.60800        },
   {104.20,        2.60800        },
   {104.25,        2.73600        },
   {104.30,        2.48000        },
   {104.34,        2.48000        },
   {104.40,        2.48000        },
   {104.45,        2.35199        },
   {104.50,        2.22400        },
   {104.55,        2.09600        },
   {104.59,        1.96800        },
   {104.65,        1.84000        },
   {104.70,        1.84000        },
   {104.75,        1.71200        },
   {104.80,        1.58400        },
   {104.84,        1.71200        },
   {104.90,        1.45600        },
   {104.95,        1.45600        },
   {105.00,        1.32800        },
   {105.05,        1.32800        },
   {105.09,        1.07200        },
   {105.15,        1.07200        },
   {105.20,       0.943999        },
   {105.25,       0.943999        },
   {105.30,       0.943999        },
   {105.34,       0.815999        },
   {105.40,       0.559999        },
   {105.45,       4.800009E-002   },
   {105.50,      -0.336000        },
   {105.55,      -0.975999        },
   {105.59,       -1.36000        },
   {105.65,       -2.00000        },
   {105.70,       -2.51200        },
   {105.75,       -2.89599        },
   {105.80,       -3.02400        },
   {105.84,       -3.02400        },
   {105.90,       -2.89599        },
   {105.95,       -2.12800        },
   {106.00,       -1.23200        },
   {106.05,      -0.464000        },
   {106.09,      -8.000003E-002   },
   {106.15,       0.303999        },
   {106.20,       0.559999        },
   {106.25,       0.687999        },
   {106.30,       0.943999        },
   {106.34,        1.07200        },
   {106.40,        1.32800        },
   {106.45,        1.45600        },
   {106.50,        1.58400        },
   {106.55,        1.58400        },
   {106.59,        1.58400        },
   {106.65,        1.58400        },
   {106.70,        1.58400        },
   {106.75,        1.45600        },
   {106.80,        1.32800        },
   {106.84,        1.45600        },
   {106.90,        1.32800        },
   {106.95,        1.32800        },
   {107.00,        1.32800        },
   {107.05,        1.20000        },
   {107.09,        1.07200        },
   {107.15,       0.943999        },
   {107.20,       0.943999        },
   {107.25,       0.815999        },
   {107.30,       0.815999        },
   {107.34,       0.687999        },
   {107.40,       0.559999        },
   {107.45,       0.559999        },
   {107.50,       0.559999        },
   {107.55,       0.431999        },
   {107.59,       0.431999        },
   {107.65,       0.559999        },
   {107.70,       0.431999        },
   {107.75,       0.687999        },
   {107.80,       0.815999        },
   {107.84,        1.07200        },
   {107.90,        1.07200        },
   {107.95,        1.32800        },
   {108.00,        1.45600        },
   {108.05,        99999.0        },
   {108.09,        99999.0        },
   {108.15,        99999.0        },
   {108.20,        99999.0        },
   {108.25,        99999.0        },
   {108.30,        99999.0        },
   {108.34,        99999.0        },
   {108.40,        99999.0        },
   {108.45,        99999.0        },
   {108.50,        99999.0        },
   {108.55,        99999.0        },
   {108.59,        99999.0        },
   {108.65,        99999.0        },
   {108.70,        99999.0        },
   {108.75,        99999.0        },
   {108.80,        99999.0        },
   {108.84,        99999.0        },
   {108.90,        99999.0        },
   {108.95,        99999.0        },
   {109.00,        99999.0        },
   {109.05,        99999.0        },
   {109.09,        99999.0        },
   {109.15,        99999.0        },
   {109.20,        99999.0        },
   {109.25,        99999.0        },
   {109.30,        99999.0        },
   {109.34,        99999.0        },
   {109.40,        99999.0        },
   {109.45,        99999.0        },
   {109.50,        99999.0        },
   {109.55,        99999.0        },
   {109.59,        99999.0        },
   {109.65,        99999.0        },
   {109.70,        99999.0        },
   {109.75,        99999.0        },
   {109.80,        99999.0        },
   {109.84,        99999.0        },
   {109.90,        99999.0        },
   {109.95,        99999.0        },
   {110.00,        99999.0        },
   {110.05,        99999.0        },
   {110.09,        99999.0        },
   {110.15,        99999.0        },
   {110.20,        99999.0        },
   {110.25,        99999.0        },
   {110.30,        99999.0        },
   {110.34,        99999.0        },
   {110.40,        99999.0        },
   {110.45,        99999.0        },
   {110.50,        99999.0        },
   {110.55,        99999.0        },
   {110.59,        99999.0        },
   {110.65,        99999.0        },
   {110.70,        99999.0        },
   {110.75,        99999.0        },
   {110.80,        99999.0        },
   {110.84,        99999.0        },
   {110.90,        99999.0        },
   {110.95,        99999.0        },
   {111.00,        99999.0        },
   {111.05,        99999.0        },
   {111.09,        99999.0        },
   {111.15,        99999.0        },
   {111.20,        99999.0        },
   {111.25,        99999.0        },
   {111.30,        99999.0        },
   {111.34,        99999.0        },
   {111.40,        99999.0        },
   {111.45,        99999.0        },
   {111.50,        99999.0        },
   {111.55,        99999.0        },
   {111.59,        99999.0        },
   {111.65,        99999.0        },
   {111.70,        99999.0        },
   {111.75,        99999.0        },
   {111.80,        99999.0        },
   {111.84,        99999.0        },
   {111.90,        99999.0        },
   {111.95,        99999.0        },
   {112.00,       4.800009E-002   },
   {112.05,      -8.000003E-002   },
   {112.09,      -8.000003E-002   },
   {112.15,      -8.000003E-002   },
   {112.20,      -8.000003E-002   },
   {112.25,      -8.000003E-002   },
   {112.30,      -0.336000        },
   {112.34,      -0.464000        },
   {112.40,      -0.336000        },
   {112.45,      -0.336000        },
   {112.50,      -0.336000        },
   {112.55,      -0.336000        },
   {112.59,      -0.336000        },
   {112.65,      -0.336000        },
   {112.70,      -8.000003E-002   },
   {112.75,       4.800009E-002   },
   {112.80,       0.176000        },
   {112.84,       0.303999        },
   {112.90,       0.559999        },
   {112.95,       0.687999        },
   {113.00,       0.943999        },
   {113.05,        1.07200        },
   {113.09,        1.20000        },
   {113.15,        1.45600        },
   {113.20,        1.71200        },
   {113.25,        1.96800        },
   {113.30,        2.35199        },
   {113.34,        2.86399        },
   {113.40,        3.12000        },
   {113.45,        3.37599        },
   {113.50,        3.63200        },
   {113.55,        3.88799        },
   {113.59,        3.88799        },
   {113.65,        3.88799        },
   {113.70,        4.01600        },
   {113.75,        3.88799        },
   {113.80,        4.01600        },
   {113.84,        3.88799        },
   {113.90,        3.75999        },
   {113.95,        3.50400        },
   {114.00,        2.99200        },
   {114.05,        2.48000        },
   {114.09,        1.96800        },
   {114.15,        1.58400        },
   {114.20,       0.943999        },
   {114.25,       0.431999        },
   {114.30,      -0.336000        },
   {114.34,      -0.847999        },
   {114.40,       -1.23200        },
   {114.45,       -1.61600        },
   {114.50,       -2.00000        },
   {114.55,       -2.25599        },
   {114.59,       -2.38399        },
   {114.65,       -2.51200        },
   {114.70,       -2.64000        },
   {114.75,       -2.89599        },
   {114.80,       -3.02400        },
   {114.84,       -2.89599        },
   {114.90,       -2.76799        },
   {114.95,       -2.64000        },
   {115.00,       -2.38399        },
   {115.05,       -2.25599        },
   {115.09,       -2.00000        },
   {115.15,       -1.74400        },
   {115.20,       -1.48800        },
   {115.25,       -1.23200        },
   {115.30,      -0.975999        },
   {115.34,      -0.719999        },
   {115.40,      -0.464000        },
   {115.45,      -0.207999        },
   {115.50,      -0.207999        },
   {115.55,       4.800009E-002   },
   {115.59,       4.800009E-002   },
   {115.65,       4.800009E-002   },
   {115.70,       4.800009E-002   },
   {115.75,      -8.000003E-002   },
   {115.80,       4.800009E-002   },
   {115.84,      -8.000003E-002   },
   {115.90,      -0.207999        },
   {115.95,      -0.464000        },
   {116.00,      -0.719999        },
   {116.05,      -0.847999        },
   {116.09,      -0.975999        },
   {116.15,       -1.10400        },
   {116.20,       -1.36000        },
   {116.25,       -1.23200        },
   {116.30,       -1.23200        },
   {116.34,       -1.23200        },
   {116.40,       -1.10400        },
   {116.45,       -1.10400        },
   {116.50,       -1.23200        },
   {116.55,       -1.10400        },
   {116.59,       -1.10400        },
   {116.65,       -1.23200        },
   {116.70,       -1.23200        },
   {116.75,       -1.36000        },
   {116.80,       -1.10400        },
   {116.84,      -0.847999        },
   {116.90,      -0.847999        },
   {116.95,       -1.10400        },
   {117.00,       -1.23200        },
   {117.05,       -1.36000        },
   {117.09,       -1.36000        },
   {117.15,       -1.23200        },
   {117.20,       -1.10400        },
   {117.25,       -1.10400        },
   {117.30,      -0.975999        },
   {117.34,      -0.847999        },
   {117.40,      -0.591999        },
   {117.45,      -0.719999        },
   {117.50,      -0.719999        },
   {117.55,      -0.591999        },
   {117.59,      -0.719999        },
   {117.65,      -0.719999        },
   {117.70,      -0.719999        },
   {117.75,      -0.719999        },
   {117.80,      -0.591999        },
   {117.84,      -0.591999        },
   {117.90,      -0.591999        },
   {117.95,      -0.464000        },
   {118.00,      -0.464000        },
   {118.05,      -0.207999        },
   {118.09,      -8.000003E-002   },
   {118.15,       4.800009E-002   },
   {118.20,      -8.000003E-002   },
   {118.25,      -8.000003E-002   },
   {118.30,      -8.000003E-002   },
   {118.34,      -8.000003E-002   },
   {118.40,      -8.000003E-002   },
   {118.45,      -8.000003E-002   },
   {118.50,      -8.000003E-002   },
   {118.55,       4.800009E-002   },
   {118.59,       4.800009E-002   },
   {118.65,       4.800009E-002   },
   {118.70,       4.800009E-002   },
   {118.75,       4.800009E-002   },
   {118.80,       4.800009E-002   },
   {118.84,       0.176000        },
   {118.90,       0.176000        },
   {118.95,       0.176000        },
   {119.00,       0.176000        },
   {119.05,      -8.000003E-002   },
   {119.09,      -0.207999        },
   {119.15,      -0.207999        },
   {119.20,      -0.207999        },
   {119.25,      -8.000003E-002   },
   {119.30,      -8.000003E-002   },
   {119.34,       4.800009E-002   },
   {119.40,       0.303999        },
   {119.45,       0.559999        },
   {119.50,       0.815999        },
   {119.55,        1.07200        },
   {119.59,        1.32800        },
   {119.65,        1.58400        },
   {119.70,        1.58400        },
   {119.75,        1.84000        },
   {119.80,        2.09600        },
   {119.84,        2.35199        },
   {119.90,        2.48000        },
   {119.95,        2.73600        },
   {120.00,        2.99200        },
   {120.05,        3.24800        },
   {120.09,        3.24800        },
   {120.15,        3.24800        },
   {120.20,        3.12000        },
   {120.25,        2.99200        },
   {120.30,        2.73600        },
   {120.34,        2.60800        },
   {120.40,        2.60800        },
   {120.45,        2.73600        },
   {120.50,        2.73600        },
   {120.55,        2.99200        },
   {120.59,        3.12000        },
   {120.65,        3.24800        },
   {120.70,        3.24800        },
   {120.75,        3.24800        },
   {120.80,        3.12000        },
   {120.84,        3.12000        },
   {120.90,        2.86399        },
   {120.95,        2.86399        },
   {121.00,        2.73600        },
   {121.05,        2.60800        },
   {121.09,        2.48000        },
   {121.15,        2.60800        },
   {121.20,        2.60800        },
   {121.25,        2.60800        },
   {121.30,        2.86399        },
   {121.34,        2.86399        },
   {121.40,        2.99200        },
   {121.45,        2.86399        },
   {121.50,        2.99200        },
   {121.55,        2.99200        },
   {121.59,        2.86399        },
   {121.65,        2.86399        },
   {121.70,        2.86399        },
   {121.75,        2.86399        },
   {121.80,        2.99200        },
   {121.84,        3.12000        },
   {121.90,        3.37599        },
   {121.95,        3.50400        },
   {122.00,        3.63200        },
   {122.05,        3.50400        },
   {122.09,        3.50400        },
   {122.15,        3.50400        },
   {122.20,        3.50400        },
   {122.25,        3.50400        },
   {122.30,        3.50400        },
   {122.34,        3.37599        },
   {122.40,        3.24800        },
   {122.45,        3.24800        },
   {122.50,        3.12000        },
   {122.55,        3.12000        },
   {122.59,        3.24800        },
   {122.65,        3.37599        },
   {122.70,        3.37599        },
   {122.75,        3.50400        },
   {122.80,        3.75999        },
   {122.84,        4.01600        },
   {122.90,        4.27200        },
   {122.95,        4.65599        },
   {123.00,        4.65599        },
   {123.05,        4.78399        },
   {123.09,        5.04000        },
   {123.15,        5.29600        },
   {123.20,        5.55199        },
   {123.25,        5.80799        },
   {123.30,        5.80799        },
   {123.34,        5.80799        },
   {123.40,        5.93599        },
   {123.45,        6.06400        },
   {123.50,        6.19200        },
   {123.55,        6.06400        },
   {123.59,        5.93599        },
   {123.65,        6.19200        },
   {123.70,        6.57599        },
   {123.75,        6.83199        },
   {123.80,        6.57599        },
   {123.84,        6.44800        },
   {123.90,        6.32000        },
   {123.95,        6.32000        },
   {124.00,        6.32000        },
   {124.05,        6.32000        },
   {124.09,        6.19200        },
   {124.15,        6.32000        },
   {124.20,        6.19200        },
   {124.25,        6.32000        },
   {124.30,        6.19200        },
   {124.34,        6.06400        },
   {124.40,        5.93599        },
   {124.45,        5.80799        },
   {124.50,        5.42400        },
   {124.55,        4.78399        },
   {124.59,        4.27200        },
   {124.65,        4.27200        },
   {124.70,        4.40000        },
   {124.75,        4.52799        },
   {124.80,        4.14400        },
   {124.84,        3.75999        },
   {124.90,        3.50400        },
   {124.95,        3.37599        },
   {125.00,        3.37599        },
   {125.05,        3.12000        },
   {125.09,        2.48000        },
   {125.15,        1.96800        },
   {125.20,        1.58400        },
   {125.25,        1.07200        },
   {125.30,       0.815999        },
   {125.34,       0.431999        },
   {125.40,       0.176000        },
   {125.45,      -8.000003E-002   },
   {125.50,      -0.207999        },
   {125.55,      -0.719999        },
   {125.59,       -1.61600        },
   {125.65,       -2.38399        },
   {125.70,       -3.15200        },
   {125.75,       -4.04800        },
   {125.80,       -4.94400        },
   {125.84,       -5.58399        },
   {125.90,       -5.96800        },
   {125.95,       -6.35200        },
   {126.00,       -6.60799        },
   {126.05,       -6.86399        },
   {126.09,       -7.88799        },
   {126.15,       -8.91200        },
   {126.20,       -9.67999        },
   {126.25,       -9.80800        },
   {126.30,       -10.1920        },
   {126.34,       -10.8320        },
   {126.40,       -11.6000        },
   {126.45,       -12.6240        },
   {126.50,       -13.6480        },
   {126.55,       -15.1839        },
   {126.59,       -17.7440        },
   {126.65,       -19.6640        },
   {126.70,       -21.3279        },
   {126.75,       -22.6080        },
   {126.80,       -23.8880        },
   {126.84,       -24.9119        },
   {126.90,       -25.8080        },
   {126.95,       -26.4480        },
   {127.00,       -26.9600        },
   {127.05,       -27.0880        },
   {127.09,       -27.6000        },
   {127.15,       -28.7519        },
   {127.20,       -29.6480        },
   {127.25,       -30.2880        },
   {127.30,       -31.3120        },
   {127.34,       -32.3359        },
   {127.40,       -32.8479        },
   {127.45,       -33.6160        },
   {127.50,       -34.6400        },
   {127.55,       -36.1760        },
   {127.59,       -37.8400        },
   {127.65,       -39.3759        },
   {127.70,       -40.6559        },
   {127.75,       -41.9360        },
   {127.80,       -42.0640        },
   {127.84,       -42.1920        },
   {127.90,       -42.0640        },
   {127.95,       -42.1920        },
   {128.00,       -42.4480        },
   {128.05,       -42.5760        },
   {128.09,       -42.7040        },
   {128.15,       -42.8320        },
   {128.19,       -43.0880        },
   {128.25,       -43.6000        },
   {128.30,       -44.3680        },
   {128.34,       -45.0080        },
   {128.40,       -45.6480        },
   {128.44,       -45.9040        },
   {128.50,       -46.0319        },
   {128.55,       -46.0319        },
   {128.59,       -46.2879        },
   {128.65,       -46.7999        },
   {128.69,       -47.1839        },
   {128.75,       -47.5679        },
   {128.80,       -47.6959        },
   {128.84,       -47.9519        },
   {128.90,       -47.8239        },
   {128.94,       -47.6959        },
   {129.00,       -47.4399        },
   {129.05,       -47.1839        },
   {129.09,       -46.9279        },
   {129.15,       -46.6719        },
   {129.19,       -46.6719        },
   {129.25,       -46.7999        },
   {129.30,       -46.5439        },
   {129.34,       -46.4159        },
   {129.40,       -46.2879        },
   {129.44,       -45.7760        },
   {129.50,       -45.3920        },
   {129.55,       -45.0080        },
   {129.59,       -44.2400        },
   {129.65,       -43.9840        },
   {129.69,       -43.4720        },
   {129.75,       -42.9600        },
   {129.80,       -42.7040        },
   {129.84,       -42.3200        },
   {129.90,       -42.0640        },
   {129.94,       -41.5520        },
   {130.00,       -40.7839        }}
share|improve this question
add comment

2 Answers 2

up vote 3 down vote accepted

You can try to fill the missing values if it is suitable for your data by:

  • mean values
  • mean values of same class
  • the most probable value
  • Spline smoothing
  • You can filter the data through a linear filter

But if you are building a nonlinear model in general, you should not detrend data before estimating nonlinear models. In the case of nonlinear grey-box models, do not detrend the data to make sure that the models represent the actual physical levels.

share|improve this answer
add comment

If your data has some cyclical behaviour, the small amount of good samples you've posted seem to have a hint of an increasing/decreasing period, and the missing data appear in blocks, you might want to consider a linear interpolation between the lst element before the bad block and the first element after.

Then add to that a random variable which has a statistical distribution similar to the rest of the data minus it's cyclical behaviour.

If nothing else looks suitable you could use a zero mean gaussian with appropriate variance as your RV.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.