# Non-linear trend reduction with missing data

How can I reduce a non-linear trend of some data series if there are parts of "no data", represented by the value 99999? For example:

data={{124.55,368.632}, {124.6,281.976}, {124.65,200.952}, {124.7,91.384}, {124.75,-60.424},
{124.8,-282.504}, {124.85,-541.448}, {124.9,-378.76}, {124.95,99999}, {125,99999},
{125.05,99999}, {125.1,99999}, {125.15,99999}, {125.2,99999}, {125.25,99999}, {125.3,99999},
{125.35,265.46}, {125.4,376.75}, {125.45,462.54},...,{178.95,284.46}}


Normally, I am successful in reducing the non-linear trend of the data via

lm = LinearModelFit[data, {a, a^2, a^3, a^4, a^5},a];
Do[aha = Last[Take[data, {m, m}]]; y = Last[aha];
x = First[aha]; new[m] = y - lm[x], {m, idim}];


but have no idea how to do the trend-reduction between those data gaps (i.e., outside of the regions with data = 99999). The position of those regions is not constant. After the reduction is performed, the position of the "no data"-values should be the same as before. Due to answer 1, unfortunately there is no periodically or cyclically behaviour in the data at all. Here, for example, is another, but complete data set:

 {{100.00,       0.303999        },
{100.05,       0.303999        },
{100.09,       0.176000        },
{100.15,       0.303999        },
{100.20,       0.176000        },
{100.25,       0.303999        },
{100.30,       0.431999        },
{100.34,       0.431999        },
{100.40,       0.559999        },
{100.45,       0.687999        },
{100.50,       0.687999        },
{100.55,       0.943999        },
{100.59,        1.07200        },
{100.65,        1.20000        },
{100.70,        1.32800        },
{100.75,        1.32800        },
{100.80,        1.20000        },
{100.84,        1.20000        },
{100.90,        1.20000        },
{100.95,        1.07200        },
{101.00,        1.07200        },
{101.05,        1.07200        },
{101.09,        1.07200        },
{101.15,        1.07200        },
{101.20,        1.07200        },
{101.25,        1.07200        },
{101.30,       0.943999        },
{101.34,       0.943999        },
{101.40,       0.943999        },
{101.45,       0.687999        },
{101.50,       0.431999        },
{101.55,       0.303999        },
{101.59,       4.800009E-002   },
{101.65,      -8.000003E-002   },
{101.70,      -0.207999        },
{101.75,      -0.464000        },
{101.80,      -0.719999        },
{101.84,      -0.847999        },
{101.90,      -0.847999        },
{101.95,      -0.975999        },
{102.00,       -1.10400        },
{102.05,       -1.23200        },
{102.09,       -1.36000        },
{102.15,       -1.36000        },
{102.20,       -1.36000        },
{102.25,       -1.36000        },
{102.30,       -1.36000        },
{102.34,       -1.23200        },
{102.40,       -1.10400        },
{102.45,       -1.10400        },
{102.50,      -0.975999        },
{102.55,      -0.847999        },
{102.59,      -0.719999        },
{102.65,      -0.591999        },
{102.70,      -0.464000        },
{102.75,      -0.207999        },
{102.80,       4.800009E-002   },
{102.84,       4.800009E-002   },
{102.90,       0.431999        },
{102.95,       0.687999        },
{103.00,       0.943999        },
{103.05,        1.20000        },
{103.09,        1.32800        },
{103.15,        1.32800        },
{103.20,        1.58400        },
{103.25,        1.71200        },
{103.30,        1.84000        },
{103.34,        1.96800        },
{103.40,        2.09600        },
{103.45,        2.22400        },
{103.50,        2.22400        },
{103.55,        2.35199        },
{103.59,        2.35199        },
{103.65,        2.48000        },
{103.70,        2.48000        },
{103.75,        2.60800        },
{103.80,        2.86399        },
{103.84,        2.86399        },
{103.90,        2.86399        },
{103.95,        2.86399        },
{104.00,        2.99200        },
{104.05,        2.86399        },
{104.09,        2.73600        },
{104.15,        2.60800        },
{104.20,        2.60800        },
{104.25,        2.73600        },
{104.30,        2.48000        },
{104.34,        2.48000        },
{104.40,        2.48000        },
{104.45,        2.35199        },
{104.50,        2.22400        },
{104.55,        2.09600        },
{104.59,        1.96800        },
{104.65,        1.84000        },
{104.70,        1.84000        },
{104.75,        1.71200        },
{104.80,        1.58400        },
{104.84,        1.71200        },
{104.90,        1.45600        },
{104.95,        1.45600        },
{105.00,        1.32800        },
{105.05,        1.32800        },
{105.09,        1.07200        },
{105.15,        1.07200        },
{105.20,       0.943999        },
{105.25,       0.943999        },
{105.30,       0.943999        },
{105.34,       0.815999        },
{105.40,       0.559999        },
{105.45,       4.800009E-002   },
{105.50,      -0.336000        },
{105.55,      -0.975999        },
{105.59,       -1.36000        },
{105.65,       -2.00000        },
{105.70,       -2.51200        },
{105.75,       -2.89599        },
{105.80,       -3.02400        },
{105.84,       -3.02400        },
{105.90,       -2.89599        },
{105.95,       -2.12800        },
{106.00,       -1.23200        },
{106.05,      -0.464000        },
{106.09,      -8.000003E-002   },
{106.15,       0.303999        },
{106.20,       0.559999        },
{106.25,       0.687999        },
{106.30,       0.943999        },
{106.34,        1.07200        },
{106.40,        1.32800        },
{106.45,        1.45600        },
{106.50,        1.58400        },
{106.55,        1.58400        },
{106.59,        1.58400        },
{106.65,        1.58400        },
{106.70,        1.58400        },
{106.75,        1.45600        },
{106.80,        1.32800        },
{106.84,        1.45600        },
{106.90,        1.32800        },
{106.95,        1.32800        },
{107.00,        1.32800        },
{107.05,        1.20000        },
{107.09,        1.07200        },
{107.15,       0.943999        },
{107.20,       0.943999        },
{107.25,       0.815999        },
{107.30,       0.815999        },
{107.34,       0.687999        },
{107.40,       0.559999        },
{107.45,       0.559999        },
{107.50,       0.559999        },
{107.55,       0.431999        },
{107.59,       0.431999        },
{107.65,       0.559999        },
{107.70,       0.431999        },
{107.75,       0.687999        },
{107.80,       0.815999        },
{107.84,        1.07200        },
{107.90,        1.07200        },
{107.95,        1.32800        },
{108.00,        1.45600        },
{108.05,        99999.0        },
{108.09,        99999.0        },
{108.15,        99999.0        },
{108.20,        99999.0        },
{108.25,        99999.0        },
{108.30,        99999.0        },
{108.34,        99999.0        },
{108.40,        99999.0        },
{108.45,        99999.0        },
{108.50,        99999.0        },
{108.55,        99999.0        },
{108.59,        99999.0        },
{108.65,        99999.0        },
{108.70,        99999.0        },
{108.75,        99999.0        },
{108.80,        99999.0        },
{108.84,        99999.0        },
{108.90,        99999.0        },
{108.95,        99999.0        },
{109.00,        99999.0        },
{109.05,        99999.0        },
{109.09,        99999.0        },
{109.15,        99999.0        },
{109.20,        99999.0        },
{109.25,        99999.0        },
{109.30,        99999.0        },
{109.34,        99999.0        },
{109.40,        99999.0        },
{109.45,        99999.0        },
{109.50,        99999.0        },
{109.55,        99999.0        },
{109.59,        99999.0        },
{109.65,        99999.0        },
{109.70,        99999.0        },
{109.75,        99999.0        },
{109.80,        99999.0        },
{109.84,        99999.0        },
{109.90,        99999.0        },
{109.95,        99999.0        },
{110.00,        99999.0        },
{110.05,        99999.0        },
{110.09,        99999.0        },
{110.15,        99999.0        },
{110.20,        99999.0        },
{110.25,        99999.0        },
{110.30,        99999.0        },
{110.34,        99999.0        },
{110.40,        99999.0        },
{110.45,        99999.0        },
{110.50,        99999.0        },
{110.55,        99999.0        },
{110.59,        99999.0        },
{110.65,        99999.0        },
{110.70,        99999.0        },
{110.75,        99999.0        },
{110.80,        99999.0        },
{110.84,        99999.0        },
{110.90,        99999.0        },
{110.95,        99999.0        },
{111.00,        99999.0        },
{111.05,        99999.0        },
{111.09,        99999.0        },
{111.15,        99999.0        },
{111.20,        99999.0        },
{111.25,        99999.0        },
{111.30,        99999.0        },
{111.34,        99999.0        },
{111.40,        99999.0        },
{111.45,        99999.0        },
{111.50,        99999.0        },
{111.55,        99999.0        },
{111.59,        99999.0        },
{111.65,        99999.0        },
{111.70,        99999.0        },
{111.75,        99999.0        },
{111.80,        99999.0        },
{111.84,        99999.0        },
{111.90,        99999.0        },
{111.95,        99999.0        },
{112.00,       4.800009E-002   },
{112.05,      -8.000003E-002   },
{112.09,      -8.000003E-002   },
{112.15,      -8.000003E-002   },
{112.20,      -8.000003E-002   },
{112.25,      -8.000003E-002   },
{112.30,      -0.336000        },
{112.34,      -0.464000        },
{112.40,      -0.336000        },
{112.45,      -0.336000        },
{112.50,      -0.336000        },
{112.55,      -0.336000        },
{112.59,      -0.336000        },
{112.65,      -0.336000        },
{112.70,      -8.000003E-002   },
{112.75,       4.800009E-002   },
{112.80,       0.176000        },
{112.84,       0.303999        },
{112.90,       0.559999        },
{112.95,       0.687999        },
{113.00,       0.943999        },
{113.05,        1.07200        },
{113.09,        1.20000        },
{113.15,        1.45600        },
{113.20,        1.71200        },
{113.25,        1.96800        },
{113.30,        2.35199        },
{113.34,        2.86399        },
{113.40,        3.12000        },
{113.45,        3.37599        },
{113.50,        3.63200        },
{113.55,        3.88799        },
{113.59,        3.88799        },
{113.65,        3.88799        },
{113.70,        4.01600        },
{113.75,        3.88799        },
{113.80,        4.01600        },
{113.84,        3.88799        },
{113.90,        3.75999        },
{113.95,        3.50400        },
{114.00,        2.99200        },
{114.05,        2.48000        },
{114.09,        1.96800        },
{114.15,        1.58400        },
{114.20,       0.943999        },
{114.25,       0.431999        },
{114.30,      -0.336000        },
{114.34,      -0.847999        },
{114.40,       -1.23200        },
{114.45,       -1.61600        },
{114.50,       -2.00000        },
{114.55,       -2.25599        },
{114.59,       -2.38399        },
{114.65,       -2.51200        },
{114.70,       -2.64000        },
{114.75,       -2.89599        },
{114.80,       -3.02400        },
{114.84,       -2.89599        },
{114.90,       -2.76799        },
{114.95,       -2.64000        },
{115.00,       -2.38399        },
{115.05,       -2.25599        },
{115.09,       -2.00000        },
{115.15,       -1.74400        },
{115.20,       -1.48800        },
{115.25,       -1.23200        },
{115.30,      -0.975999        },
{115.34,      -0.719999        },
{115.40,      -0.464000        },
{115.45,      -0.207999        },
{115.50,      -0.207999        },
{115.55,       4.800009E-002   },
{115.59,       4.800009E-002   },
{115.65,       4.800009E-002   },
{115.70,       4.800009E-002   },
{115.75,      -8.000003E-002   },
{115.80,       4.800009E-002   },
{115.84,      -8.000003E-002   },
{115.90,      -0.207999        },
{115.95,      -0.464000        },
{116.00,      -0.719999        },
{116.05,      -0.847999        },
{116.09,      -0.975999        },
{116.15,       -1.10400        },
{116.20,       -1.36000        },
{116.25,       -1.23200        },
{116.30,       -1.23200        },
{116.34,       -1.23200        },
{116.40,       -1.10400        },
{116.45,       -1.10400        },
{116.50,       -1.23200        },
{116.55,       -1.10400        },
{116.59,       -1.10400        },
{116.65,       -1.23200        },
{116.70,       -1.23200        },
{116.75,       -1.36000        },
{116.80,       -1.10400        },
{116.84,      -0.847999        },
{116.90,      -0.847999        },
{116.95,       -1.10400        },
{117.00,       -1.23200        },
{117.05,       -1.36000        },
{117.09,       -1.36000        },
{117.15,       -1.23200        },
{117.20,       -1.10400        },
{117.25,       -1.10400        },
{117.30,      -0.975999        },
{117.34,      -0.847999        },
{117.40,      -0.591999        },
{117.45,      -0.719999        },
{117.50,      -0.719999        },
{117.55,      -0.591999        },
{117.59,      -0.719999        },
{117.65,      -0.719999        },
{117.70,      -0.719999        },
{117.75,      -0.719999        },
{117.80,      -0.591999        },
{117.84,      -0.591999        },
{117.90,      -0.591999        },
{117.95,      -0.464000        },
{118.00,      -0.464000        },
{118.05,      -0.207999        },
{118.09,      -8.000003E-002   },
{118.15,       4.800009E-002   },
{118.20,      -8.000003E-002   },
{118.25,      -8.000003E-002   },
{118.30,      -8.000003E-002   },
{118.34,      -8.000003E-002   },
{118.40,      -8.000003E-002   },
{118.45,      -8.000003E-002   },
{118.50,      -8.000003E-002   },
{118.55,       4.800009E-002   },
{118.59,       4.800009E-002   },
{118.65,       4.800009E-002   },
{118.70,       4.800009E-002   },
{118.75,       4.800009E-002   },
{118.80,       4.800009E-002   },
{118.84,       0.176000        },
{118.90,       0.176000        },
{118.95,       0.176000        },
{119.00,       0.176000        },
{119.05,      -8.000003E-002   },
{119.09,      -0.207999        },
{119.15,      -0.207999        },
{119.20,      -0.207999        },
{119.25,      -8.000003E-002   },
{119.30,      -8.000003E-002   },
{119.34,       4.800009E-002   },
{119.40,       0.303999        },
{119.45,       0.559999        },
{119.50,       0.815999        },
{119.55,        1.07200        },
{119.59,        1.32800        },
{119.65,        1.58400        },
{119.70,        1.58400        },
{119.75,        1.84000        },
{119.80,        2.09600        },
{119.84,        2.35199        },
{119.90,        2.48000        },
{119.95,        2.73600        },
{120.00,        2.99200        },
{120.05,        3.24800        },
{120.09,        3.24800        },
{120.15,        3.24800        },
{120.20,        3.12000        },
{120.25,        2.99200        },
{120.30,        2.73600        },
{120.34,        2.60800        },
{120.40,        2.60800        },
{120.45,        2.73600        },
{120.50,        2.73600        },
{120.55,        2.99200        },
{120.59,        3.12000        },
{120.65,        3.24800        },
{120.70,        3.24800        },
{120.75,        3.24800        },
{120.80,        3.12000        },
{120.84,        3.12000        },
{120.90,        2.86399        },
{120.95,        2.86399        },
{121.00,        2.73600        },
{121.05,        2.60800        },
{121.09,        2.48000        },
{121.15,        2.60800        },
{121.20,        2.60800        },
{121.25,        2.60800        },
{121.30,        2.86399        },
{121.34,        2.86399        },
{121.40,        2.99200        },
{121.45,        2.86399        },
{121.50,        2.99200        },
{121.55,        2.99200        },
{121.59,        2.86399        },
{121.65,        2.86399        },
{121.70,        2.86399        },
{121.75,        2.86399        },
{121.80,        2.99200        },
{121.84,        3.12000        },
{121.90,        3.37599        },
{121.95,        3.50400        },
{122.00,        3.63200        },
{122.05,        3.50400        },
{122.09,        3.50400        },
{122.15,        3.50400        },
{122.20,        3.50400        },
{122.25,        3.50400        },
{122.30,        3.50400        },
{122.34,        3.37599        },
{122.40,        3.24800        },
{122.45,        3.24800        },
{122.50,        3.12000        },
{122.55,        3.12000        },
{122.59,        3.24800        },
{122.65,        3.37599        },
{122.70,        3.37599        },
{122.75,        3.50400        },
{122.80,        3.75999        },
{122.84,        4.01600        },
{122.90,        4.27200        },
{122.95,        4.65599        },
{123.00,        4.65599        },
{123.05,        4.78399        },
{123.09,        5.04000        },
{123.15,        5.29600        },
{123.20,        5.55199        },
{123.25,        5.80799        },
{123.30,        5.80799        },
{123.34,        5.80799        },
{123.40,        5.93599        },
{123.45,        6.06400        },
{123.50,        6.19200        },
{123.55,        6.06400        },
{123.59,        5.93599        },
{123.65,        6.19200        },
{123.70,        6.57599        },
{123.75,        6.83199        },
{123.80,        6.57599        },
{123.84,        6.44800        },
{123.90,        6.32000        },
{123.95,        6.32000        },
{124.00,        6.32000        },
{124.05,        6.32000        },
{124.09,        6.19200        },
{124.15,        6.32000        },
{124.20,        6.19200        },
{124.25,        6.32000        },
{124.30,        6.19200        },
{124.34,        6.06400        },
{124.40,        5.93599        },
{124.45,        5.80799        },
{124.50,        5.42400        },
{124.55,        4.78399        },
{124.59,        4.27200        },
{124.65,        4.27200        },
{124.70,        4.40000        },
{124.75,        4.52799        },
{124.80,        4.14400        },
{124.84,        3.75999        },
{124.90,        3.50400        },
{124.95,        3.37599        },
{125.00,        3.37599        },
{125.05,        3.12000        },
{125.09,        2.48000        },
{125.15,        1.96800        },
{125.20,        1.58400        },
{125.25,        1.07200        },
{125.30,       0.815999        },
{125.34,       0.431999        },
{125.40,       0.176000        },
{125.45,      -8.000003E-002   },
{125.50,      -0.207999        },
{125.55,      -0.719999        },
{125.59,       -1.61600        },
{125.65,       -2.38399        },
{125.70,       -3.15200        },
{125.75,       -4.04800        },
{125.80,       -4.94400        },
{125.84,       -5.58399        },
{125.90,       -5.96800        },
{125.95,       -6.35200        },
{126.00,       -6.60799        },
{126.05,       -6.86399        },
{126.09,       -7.88799        },
{126.15,       -8.91200        },
{126.20,       -9.67999        },
{126.25,       -9.80800        },
{126.30,       -10.1920        },
{126.34,       -10.8320        },
{126.40,       -11.6000        },
{126.45,       -12.6240        },
{126.50,       -13.6480        },
{126.55,       -15.1839        },
{126.59,       -17.7440        },
{126.65,       -19.6640        },
{126.70,       -21.3279        },
{126.75,       -22.6080        },
{126.80,       -23.8880        },
{126.84,       -24.9119        },
{126.90,       -25.8080        },
{126.95,       -26.4480        },
{127.00,       -26.9600        },
{127.05,       -27.0880        },
{127.09,       -27.6000        },
{127.15,       -28.7519        },
{127.20,       -29.6480        },
{127.25,       -30.2880        },
{127.30,       -31.3120        },
{127.34,       -32.3359        },
{127.40,       -32.8479        },
{127.45,       -33.6160        },
{127.50,       -34.6400        },
{127.55,       -36.1760        },
{127.59,       -37.8400        },
{127.65,       -39.3759        },
{127.70,       -40.6559        },
{127.75,       -41.9360        },
{127.80,       -42.0640        },
{127.84,       -42.1920        },
{127.90,       -42.0640        },
{127.95,       -42.1920        },
{128.00,       -42.4480        },
{128.05,       -42.5760        },
{128.09,       -42.7040        },
{128.15,       -42.8320        },
{128.19,       -43.0880        },
{128.25,       -43.6000        },
{128.30,       -44.3680        },
{128.34,       -45.0080        },
{128.40,       -45.6480        },
{128.44,       -45.9040        },
{128.50,       -46.0319        },
{128.55,       -46.0319        },
{128.59,       -46.2879        },
{128.65,       -46.7999        },
{128.69,       -47.1839        },
{128.75,       -47.5679        },
{128.80,       -47.6959        },
{128.84,       -47.9519        },
{128.90,       -47.8239        },
{128.94,       -47.6959        },
{129.00,       -47.4399        },
{129.05,       -47.1839        },
{129.09,       -46.9279        },
{129.15,       -46.6719        },
{129.19,       -46.6719        },
{129.25,       -46.7999        },
{129.30,       -46.5439        },
{129.34,       -46.4159        },
{129.40,       -46.2879        },
{129.44,       -45.7760        },
{129.50,       -45.3920        },
{129.55,       -45.0080        },
{129.59,       -44.2400        },
{129.65,       -43.9840        },
{129.69,       -43.4720        },
{129.75,       -42.9600        },
{129.80,       -42.7040        },
{129.84,       -42.3200        },
{129.90,       -42.0640        },
{129.94,       -41.5520        },
{130.00,       -40.7839        }}


You can try to fill the missing values if it is suitable for your data by:

• mean values
• mean values of same class
• the most probable value
• Spline smoothing
• You can filter the data through a linear filter

But if you are building a nonlinear model in general, you should not detrend data before estimating nonlinear models. In the case of nonlinear grey-box models, do not detrend the data to make sure that the models represent the actual physical levels.

If your data has some cyclical behaviour, the small amount of good samples you've posted seem to have a hint of an increasing/decreasing period, and the missing data appear in blocks, you might want to consider a linear interpolation between the lst element before the bad block and the first element after.

Then add to that a random variable which has a statistical distribution similar to the rest of the data minus it's cyclical behaviour.

If nothing else looks suitable you could use a zero mean gaussian with appropriate variance as your RV.