3
$\begingroup$

I have this data shown in picture 1 below and I want to filter it. I tried some filters but I couldn't get good results. seems that I didn't choose the best parameters for it.

noisy data

I want to smooth my data (in blue) to be closer to the one in red (without using the data of the red curve.) I appreciate any help.

after smoothing


This is the list I am using in a CSV file (sorry I am a new user in StackExchange and don't know how to upload CSV file here)

data={{-0.198,-0.0934245},{-0.196,-0.17041},{-0.194,-0.0829264},{-0.192,0.134033},{-0.19,0.134033},{-0.188,-0.110921},{-0.186,-0.156413},{-0.184,-0.0479329},{-0.182,0.172526},{-0.18,0.0605469},{-0.178,-0.00244141},{-0.176,0.00105794},{-0.174,-0.128418},{-0.172,-0.058431},{-0.17,-0.068929},{-0.168,0.0955404},{-0.166,0.120036},{-0.164,0.20402},{-0.162,0.0570475},{-0.16,0.0990397},{-0.158,0.270508},{-0.156,-0.0759277},{-0.154,0.158529},{-0.152,0.144531},{-0.15,0.253011},{-0.148,0.15153},{-0.146,0.315999},{-0.144,0.291504},{-0.142,0.214518},{-0.14,0.169027},{-0.138,0.25651},{-0.136,0.176025},{-0.134,0.232015},{-0.132,0.197021},{-0.13,0.081543},{-0.128,0.00105794},{-0.126,0.221517},{-0.124,0.0780436},{-0.122,0.214518},{-0.12,0.190023},{-0.118,0.413981},{-0.116,0.134033},{-0.114,0.211019},{-0.112,0.0955404},{-0.11,0.123535},{-0.108,0.0745443},{-0.106,0.228516},{-0.104,0.162028},{-0.102,0.162028},{-0.1,-0.0409342},{-0.098,0.190023},{-0.096,0.158529},{-0.094,0.295003},{-0.092,0.315999},{-0.09,0.270508},{-0.088,0.197021},{-0.086,0.281006},{-0.084,0.25651},{-0.082,0.158529},{-0.08,0.20752},{-0.078,0.281006},{-0.076,0.322998},{-0.074,0.239014},{-0.072,0.148031},{-0.07,0.424479},{-0.068,0.281006},{-0.066,0.295003},{-0.064,0.270508},{-0.062,0.130534},{-0.06,0.459473},{-0.058,0.36499},{-0.056,0.242513},{-0.054,0.113037},{-0.052,0.438477},{-0.05,0.165527},{-0.048,0.281006},{-0.046,0.148031},{-0.044,0.267008},{-0.042,0.155029},{-0.04,0.274007},{-0.038,0.315999},{-0.036,0.445475},{-0.034,0.109538},{-0.032,0.20752},{-0.03,0.319499},{-0.028,0.218018},{-0.026,0.158529},{-0.024,0.20752},{-0.022,0.221517},{-0.02,0.158529},{-0.018,0.291504},{-0.016,0.25651},{-0.014,0.25651},{-0.012,0.281006},{-0.01,0.441976},{-0.008,0.106038},{-0.006,0.214518},{-0.004,0.354492},{-0.002,0.288005},{1.53*10^-16,0.274007},{0.002,0.197021},{0.004,0.200521},{0.006,0.092041},{0.008,0.232015},{0.01,0.347493},{0.012,0.249512},{0.014,0.497965},{0.016,0.406982},{0.018,0.193522},{0.02,0.291504},{0.022,0.288005},{0.024,0.232015},{0.026,0.169027},{0.028,0.424479},{0.03,0.081543},{0.032,0.249512},{0.034,0.378988},{0.036,0.221517},{0.038,0.253011},{0.04,0.336995},{0.042,0.483968},{0.044,0.424479},{0.046,0.336995},{0.048,0.47347},{0.05,0.375488},{0.052,0.42098},{0.054,0.42098},{0.056,0.357992},{0.058,0.378988},{0.06,0.476969},{0.062,0.235514},{0.064,0.340495},{0.066,0.396484},{0.068,0.357992},{0.07,0.490967},{0.072,0.263509},{0.074,0.263509},{0.076,0.225016},{0.078,0.36499},{0.08,0.267008},{0.082,0.179525},{0.084,0.211019},{0.086,0.382487},{0.088,0.340495},{0.09,0.319499},{0.092,0.459473},{0.094,0.288005},{0.096,0.350993},{0.098,0.116536},{0.1,0.309001},{0.102,0.277507},{0.104,0.497965},{0.106,0.36499},{0.108,0.225016},{0.11,0.134033},{0.112,0.315999},{0.114,0.47347},{0.116,0.336995},{0.118,0.448975},{0.12,0.281006},{0.122,0.298503},{0.124,0.410482},{0.126,0.3125},{0.128,0.336995},{0.13,0.403483},{0.132,0.326497},{0.134,0.399984},{0.136,0.36849},{0.138,0.560954},{0.14,0.522461},{0.142,0.221517},{0.144,0.494466},{0.146,0.385986},{0.148,0.326497},{0.15,0.291504},{0.152,0.26001},{0.154,0.309001},{0.156,0.36499},{0.158,0.455973},{0.16,0.455973},{0.162,0.378988},{0.164,0.41748},{0.166,0.225016},{0.168,0.183024},{0.17,0.26001},{0.172,0.63444},{0.174,0.176025},{0.176,0.455973},{0.178,0.267008},{0.18,0.396484},{0.182,0.504964},{0.184,0.277507},{0.186,0.246012},{0.188,0.427979},{0.19,0.543457},{0.192,0.518962},{0.194,0.511963},{0.196,0.295003},{0.198,0.511963},{0.2,0.613444},{0.202,0.487467},{0.204,0.263509},{0.206,0.36499},{0.208,0.357992},{0.21,0.242513},{0.212,0.211019},{0.214,0.431478},{0.216,0.354492},{0.218,0.431478},{0.22,0.466471},{0.222,0.406982},{0.224,0.350993},{0.226,0.560954},{0.228,0.389486},{0.23,0.52946},{0.232,0.550456},{0.234,0.267008},{0.236,0.518962},{0.238,0.515462},{0.24,0.42098},{0.242,0.550456},{0.244,0.350993},{0.246,0.522461},{0.248,0.424479},{0.25,0.424479},{0.252,0.284505},{0.254,0.63444},{0.256,0.637939},{0.258,0.326497},{0.26,0.585449},{0.262,0.375488},{0.264,0.515462},{0.266,0.406982},{0.268,0.466471},{0.27,0.487467},{0.272,0.560954},{0.274,0.47347},{0.276,0.438477},{0.278,0.543457},{0.28,0.424479},{0.282,0.721924},{0.284,0.396484},{0.286,0.434977},{0.288,0.553955},{0.29,0.518962},{0.292,0.644938},{0.294,0.63444},{0.296,0.438477},{0.298,0.47347},{0.3,0.679932},{0.302,0.522461},{0.304,0.487467},{0.306,0.900391},{0.308,0.616943},{0.31,0.539958},{0.312,0.462972},{0.314,0.648438},{0.316,0.543457},{0.318,0.756917},{0.32,0.683431},{0.322,0.697428},{0.324,0.781413},{0.326,0.79541},{0.328,0.900391},{0.33,0.69043},{0.332,0.819906},{0.334,0.718424},{0.336,0.739421},{0.338,0.714925},{0.34,0.714925},{0.342,0.753418},{0.344,0.721924},{0.346,0.854899},{0.348,0.74292},{0.35,1.03687},{0.352,1.01237},{0.354,0.942383},{0.356,1.07886},{0.358,0.8479},{0.36,0.942383},{0.362,1.01587},{0.364,0.721924},{0.366,1.07886},{0.368,0.882894},{0.37,0.840902},{0.372,0.987874},{0.374,0.970378},{0.376,0.924886},{0.378,0.910889},{0.38,0.8514},{0.382,0.935384},{0.384,1.11385},{0.386,1.05786},{0.388,1.10685},{0.39,1.16634},{0.392,0.700928},{0.394,1.16634},{0.396,1.21883},{0.398,1.12085},{0.4,1.05436},{0.398,1.31681},{0.396,1.12435},{0.394,1.00887},{0.392,1.14884},{0.39,1.56527},{0.388,0.693929},{0.386,0.483968},{0.384,0.186523},{0.382,0.410482},{0.38,0.211019},{0.378,0.295003},{0.376,0.343994},{0.374,0.20752},{0.372,-0.0164388},{0.37,0.357992},{0.368,0.357992},{0.366,0.0255534},{0.364,-0.0199382},{0.362,-0.00594076},{0.36,0.0605469},{0.358,-0.212402},{0.356,-0.0479329},{0.354,-0.058431},{0.352,-0.289388},{0.35,-0.0199382},{0.348,-0.292887},{0.346,-0.191406},{0.344,-0.310384},{0.342,-0.166911},{0.34,-0.348877},{0.338,-0.282389},{0.336,-0.0444336},{0.334,-0.380371},{0.332,-0.289388},{0.33,-0.166911},{0.328,-0.471354},{0.326,-0.33488},{0.324,-0.38737},{0.322,-0.250895},{0.32,-0.271891},{0.318,-0.205404},{0.316,-0.446859},{0.314,-0.390869},{0.312,-0.299886},{0.31,-0.33488},{0.308,-0.341878},{0.306,-0.320882},{0.304,-0.418864},{0.302,-0.362874},{0.3,-0.551839},{0.298,-0.625326},{0.296,-0.313883},{0.294,-0.436361},{0.292,-0.436361},{0.29,-0.247396},{0.288,-0.341878},{0.286,-0.149414},{0.284,-0.453857},{0.282,-0.352376},{0.28,-0.296387},{0.278,-0.149414},{0.276,-0.149414},{0.274,-0.0409342},{0.272,-0.27889},{0.27,-0.429362},{0.268,-0.0374349},{0.266,-0.0829264},{0.264,-0.289388},{0.262,-0.607829},{0.26,-0.359375},{0.258,-0.240397},{0.256,-0.194906},{0.254,-0.156413},{0.252,-0.33138},{0.25,-0.488851},{0.248,-0.320882},{0.246,-0.292887},{0.244,-0.394368},{0.242,-0.446859},{0.24,-0.0619303},{0.238,-0.0654297},{0.236,-0.114421},{0.234,-0.509847},{0.232,-0.103923},{0.23,-0.152913},{0.228,-0.128418},{0.226,-0.450358},{0.224,-0.296387},{0.222,-0.268392},{0.22,-0.369873},{0.218,-0.0724284},{0.216,-0.219401},{0.214,-0.264893},{0.212,-0.285889},{0.21,-0.348877},{0.208,-0.485352},{0.206,-0.219401},{0.204,-0.0864258},{0.202,-0.282389},{0.2,-0.366374},{0.198,-0.443359},{0.196,-0.180908},{0.194,-0.159912},{0.192,-0.310384},{0.19,-0.460856},{0.188,-0.247396},{0.186,-0.27889},{0.184,-0.27889},{0.182,-0.516846},{0.18,-0.366374},{0.178,-0.285889},{0.176,-0.296387},{0.174,-0.166911},{0.172,-0.264893},{0.17,-0.285889},{0.168,-0.33488},{0.166,-0.229899},{0.164,-0.303385},{0.162,-0.268392},{0.16,-0.156413},{0.158,-0.390869},{0.156,-0.341878},{0.154,-0.159912},{0.152,-0.397868},{0.15,-0.236898},{0.148,-0.394368},{0.146,-0.33138},{0.144,-0.348877},{0.142,-0.369873},{0.14,-0.17041},{0.138,-0.394368},{0.136,-0.338379},{0.134,-0.268392},{0.132,-0.306885},{0.13,-0.2229},{0.128,-0.275391},{0.126,-0.299886},{0.124,-0.247396},{0.122,-0.327881},{0.12,-0.418864},{0.118,-0.394368},{0.116,-0.156413},{0.114,-0.380371},{0.112,-0.352376},{0.11,-0.135417},{0.108,-0.397868},{0.106,-0.404867},{0.104,-0.397868},{0.102,-0.33138},{0.1,-0.17391},{0.098,-0.345378},{0.096,-0.348877},{0.094,-0.348877},{0.092,-0.355876},{0.09,-0.366374},{0.088,-0.436361},{0.086,-0.359375},{0.084,-0.292887},{0.082,-0.380371},{0.08,-0.534342},{0.078,-0.285889},{0.076,-0.33138},{0.074,-0.327881},{0.072,-0.180908},{0.07,-0.268392},{0.068,-0.310384},{0.066,-0.352376},{0.064,-0.436361},{0.062,-0.229899},{0.06,-0.292887},{0.058,-0.338379},{0.056,-0.373372},{0.054,-0.513346},{0.052,-0.338379},{0.05,-0.240397},{0.048,-0.275391},{0.046,-0.513346},{0.044,-0.446859},{0.042,-0.338379},{0.04,-0.268392},{0.038,-0.320882},{0.036,-0.219401},{0.034,-0.432861},{0.032,-0.317383},{0.03,-0.376872},{0.028,-0.471354},{0.026,-0.345378},{0.024,-0.366374},{0.022,-0.401367},{0.02,-0.527344},{0.018,-0.485352},{0.016,-0.271891},{0.014,-0.27889},{0.012,-0.453857},{0.01,-0.527344},{0.008,-0.366374},{0.006,-0.366374},{0.004,-0.194906},{0.002,-0.369873},{1.25*10^-16,-0.485352},{-0.002,-0.597331},{-0.004,-0.338379},{-0.006,-0.380371},{-0.008,-0.394368},{-0.01,-0.443359},{-0.012,-0.285889},{-0.014,-0.436361},{-0.016,-0.516846},{-0.018,-0.418864},{-0.02,-0.338379},{-0.022,-0.320882},{-0.024,-0.317383},{-0.026,-0.33488},{-0.028,-0.327881},{-0.03,-0.2264},{-0.032,-0.317383},{-0.034,-0.359375},{-0.036,-0.38387},{-0.038,-0.576335},{-0.04,-0.677816},{-0.042,-0.418864},{-0.044,-0.558838},{-0.046,-0.450358},{-0.048,-0.499349},{-0.05,-0.352376},{-0.052,-0.527344},{-0.054,-0.159912},{-0.056,-0.523844},{-0.058,-0.257894},{-0.06,-0.460856},{-0.062,-0.369873},{-0.064,-0.397868},{-0.066,-0.614827},{-0.068,-0.516846},{-0.07,-0.257894},{-0.072,-0.558838},{-0.074,-0.415365},{-0.076,-0.730306},{-0.078,-0.296387},{-0.08,-0.523844},{-0.082,-0.49585},{-0.084,-0.502848},{-0.086,-0.562337},{-0.088,-0.562337},{-0.09,-0.310384},{-0.092,-0.373372},{-0.094,-0.373372},{-0.096,-0.320882},{-0.098,-0.380371},{-0.1,-0.530843},{-0.102,-0.397868},{-0.104,-0.572835},{-0.106,-0.516846},{-0.108,-0.408366},{-0.11,-0.380371},{-0.112,-0.527344},{-0.114,-0.478353},{-0.116,-0.579834},{-0.118,-0.425863},{-0.12,-0.310384},{-0.122,-0.457357},{-0.124,-0.520345},{-0.126,-0.338379},{-0.128,-0.27889},{-0.13,-0.579834},{-0.132,-0.555339},{-0.134,-0.593831},{-0.136,-0.541341},{-0.138,-0.464355},{-0.14,-0.534342},{-0.142,-0.450358},{-0.144,-0.432861},{-0.146,-0.366374},{-0.148,-0.478353},{-0.15,-0.443359},{-0.152,-0.488851},{-0.154,-0.681315},{-0.156,-0.635824},{-0.158,-0.345378},{-0.16,-0.49585},{-0.162,-0.597331},{-0.164,-0.502848},{-0.166,-0.485352},{-0.168,-0.621826},{-0.17,-0.516846},{-0.172,-0.625326},{-0.174,-0.572835},{-0.176,-0.478353},{-0.178,-0.569336},{-0.18,-0.569336},{-0.182,-0.569336},{-0.184,-0.712809},{-0.186,-0.723307},{-0.188,-0.569336},{-0.19,-0.586833},{-0.192,-0.604329},{-0.194,-0.674316},{-0.196,-0.688314},{-0.198,-0.614827},{-0.2,-0.747803}
};

I used this code and tried to change the parameters but couldn't get the right ones for a smooth curve.

ListLinePlot[{data, 
 BilateralFilter[data, 2, .14, MaxIterations -> 5]}, 
 PlotStyle -> {Thin, Red}]

I tried to follow this example: Remove noise from data How can we choose the right parameters? Thanks,

$\endgroup$
  • 2
    $\begingroup$ Whatever your data, I'd suggest looking at reference.wolframcloud.com/language/guide/… - possibly first converting your data to TimeSeries. $\endgroup$ – kirma Jul 10 '18 at 10:55
  • $\begingroup$ Thanks for providing data, what about the code that failed you? $\endgroup$ – Kuba Jul 10 '18 at 12:13
  • $\begingroup$ Is the noise vertical, or should both dimensions be treated as noisy? $\endgroup$ – lasen H Jul 10 '18 at 13:20
  • $\begingroup$ yes the noise is actually vertical $\endgroup$ – Mat-laut Jul 10 '18 at 13:47
3
$\begingroup$

An approach using FindFormula

data = {{-0.198, -0.0934245}, {-0.196, -0.17041}, {-0.194, -0.0829264}, \
{-0.192, 0.134033}, {-0.19, 
    0.134033}, {-0.188, -0.110921}, {-0.186, -0.156413}, {-0.184, \
-0.0479329}, {-0.182, 0.172526}, {-0.18, 
    0.0605469}, {-0.178, -0.00244141}, {-0.176, 
    0.00105794}, {-0.174, -0.128418}, {-0.172, -0.058431}, {-0.17, \
-0.068929}, {-0.168, 0.0955404}, {-0.166, 0.120036}, {-0.164, 
    0.20402}, {-0.162, 0.0570475}, {-0.16, 0.0990397}, {-0.158, 
    0.270508}, {-0.156, -0.0759277}, {-0.154, 0.158529}, {-0.152, 
    0.144531}, {-0.15, 0.253011}, {-0.148, 0.15153}, {-0.146, 
    0.315999}, {-0.144, 0.291504}, {-0.142, 0.214518}, {-0.14, 
    0.169027}, {-0.138, 0.25651}, {-0.136, 0.176025}, {-0.134, 
    0.232015}, {-0.132, 0.197021}, {-0.13, 0.081543}, {-0.128, 
    0.00105794}, {-0.126, 0.221517}, {-0.124, 0.0780436}, {-0.122, 
    0.214518}, {-0.12, 0.190023}, {-0.118, 0.413981}, {-0.116, 
    0.134033}, {-0.114, 0.211019}, {-0.112, 0.0955404}, {-0.11, 
    0.123535}, {-0.108, 0.0745443}, {-0.106, 0.228516}, {-0.104, 
    0.162028}, {-0.102, 0.162028}, {-0.1, -0.0409342}, {-0.098, 
    0.190023}, {-0.096, 0.158529}, {-0.094, 0.295003}, {-0.092, 
    0.315999}, {-0.09, 0.270508}, {-0.088, 0.197021}, {-0.086, 
    0.281006}, {-0.084, 0.25651}, {-0.082, 0.158529}, {-0.08, 
    0.20752}, {-0.078, 0.281006}, {-0.076, 0.322998}, {-0.074, 
    0.239014}, {-0.072, 0.148031}, {-0.07, 0.424479}, {-0.068, 
    0.281006}, {-0.066, 0.295003}, {-0.064, 0.270508}, {-0.062, 
    0.130534}, {-0.06, 0.459473}, {-0.058, 0.36499}, {-0.056, 
    0.242513}, {-0.054, 0.113037}, {-0.052, 0.438477}, {-0.05, 
    0.165527}, {-0.048, 0.281006}, {-0.046, 0.148031}, {-0.044, 
    0.267008}, {-0.042, 0.155029}, {-0.04, 0.274007}, {-0.038, 
    0.315999}, {-0.036, 0.445475}, {-0.034, 0.109538}, {-0.032, 
    0.20752}, {-0.03, 0.319499}, {-0.028, 0.218018}, {-0.026, 
    0.158529}, {-0.024, 0.20752}, {-0.022, 0.221517}, {-0.02, 
    0.158529}, {-0.018, 0.291504}, {-0.016, 0.25651}, {-0.014, 
    0.25651}, {-0.012, 0.281006}, {-0.01, 0.441976}, {-0.008, 
    0.106038}, {-0.006, 0.214518}, {-0.004, 0.354492}, {-0.002, 
    0.288005}, {1.53*10^-16, 0.274007}, {0.002, 0.197021}, {0.004, 
    0.200521}, {0.006, 0.092041}, {0.008, 0.232015}, {0.01, 0.347493}, {0.012,
     0.249512}, {0.014, 0.497965}, {0.016, 0.406982}, {0.018, 
    0.193522}, {0.02, 0.291504}, {0.022, 0.288005}, {0.024, 0.232015}, {0.026,
     0.169027}, {0.028, 0.424479}, {0.03, 0.081543}, {0.032, 
    0.249512}, {0.034, 0.378988}, {0.036, 0.221517}, {0.038, 0.253011}, {0.04,
     0.336995}, {0.042, 0.483968}, {0.044, 0.424479}, {0.046, 
    0.336995}, {0.048, 0.47347}, {0.05, 0.375488}, {0.052, 0.42098}, {0.054, 
    0.42098}, {0.056, 0.357992}, {0.058, 0.378988}, {0.06, 0.476969}, {0.062, 
    0.235514}, {0.064, 0.340495}, {0.066, 0.396484}, {0.068, 0.357992}, {0.07,
     0.490967}, {0.072, 0.263509}, {0.074, 0.263509}, {0.076, 
    0.225016}, {0.078, 0.36499}, {0.08, 0.267008}, {0.082, 0.179525}, {0.084, 
    0.211019}, {0.086, 0.382487}, {0.088, 0.340495}, {0.09, 0.319499}, {0.092,
     0.459473}, {0.094, 0.288005}, {0.096, 0.350993}, {0.098, 0.116536}, {0.1,
     0.309001}, {0.102, 0.277507}, {0.104, 0.497965}, {0.106, 
    0.36499}, {0.108, 0.225016}, {0.11, 0.134033}, {0.112, 0.315999}, {0.114, 
    0.47347}, {0.116, 0.336995}, {0.118, 0.448975}, {0.12, 0.281006}, {0.122, 
    0.298503}, {0.124, 0.410482}, {0.126, 0.3125}, {0.128, 0.336995}, {0.13, 
    0.403483}, {0.132, 0.326497}, {0.134, 0.399984}, {0.136, 0.36849}, {0.138,
     0.560954}, {0.14, 0.522461}, {0.142, 0.221517}, {0.144, 
    0.494466}, {0.146, 0.385986}, {0.148, 0.326497}, {0.15, 0.291504}, {0.152,
     0.26001}, {0.154, 0.309001}, {0.156, 0.36499}, {0.158, 0.455973}, {0.16, 
    0.455973}, {0.162, 0.378988}, {0.164, 0.41748}, {0.166, 0.225016}, {0.168,
     0.183024}, {0.17, 0.26001}, {0.172, 0.63444}, {0.174, 0.176025}, {0.176, 
    0.455973}, {0.178, 0.267008}, {0.18, 0.396484}, {0.182, 0.504964}, {0.184,
     0.277507}, {0.186, 0.246012}, {0.188, 0.427979}, {0.19, 
    0.543457}, {0.192, 0.518962}, {0.194, 0.511963}, {0.196, 
    0.295003}, {0.198, 0.511963}, {0.2, 0.613444}, {0.202, 0.487467}, {0.204, 
    0.263509}, {0.206, 0.36499}, {0.208, 0.357992}, {0.21, 0.242513}, {0.212, 
    0.211019}, {0.214, 0.431478}, {0.216, 0.354492}, {0.218, 0.431478}, {0.22,
     0.466471}, {0.222, 0.406982}, {0.224, 0.350993}, {0.226, 
    0.560954}, {0.228, 0.389486}, {0.23, 0.52946}, {0.232, 0.550456}, {0.234, 
    0.267008}, {0.236, 0.518962}, {0.238, 0.515462}, {0.24, 0.42098}, {0.242, 
    0.550456}, {0.244, 0.350993}, {0.246, 0.522461}, {0.248, 0.424479}, {0.25,
     0.424479}, {0.252, 0.284505}, {0.254, 0.63444}, {0.256, 
    0.637939}, {0.258, 0.326497}, {0.26, 0.585449}, {0.262, 0.375488}, {0.264,
     0.515462}, {0.266, 0.406982}, {0.268, 0.466471}, {0.27, 
    0.487467}, {0.272, 0.560954}, {0.274, 0.47347}, {0.276, 0.438477}, {0.278,
     0.543457}, {0.28, 0.424479}, {0.282, 0.721924}, {0.284, 
    0.396484}, {0.286, 0.434977}, {0.288, 0.553955}, {0.29, 0.518962}, {0.292,
     0.644938}, {0.294, 0.63444}, {0.296, 0.438477}, {0.298, 0.47347}, {0.3, 
    0.679932}, {0.302, 0.522461}, {0.304, 0.487467}, {0.306, 
    0.900391}, {0.308, 0.616943}, {0.31, 0.539958}, {0.312, 0.462972}, {0.314,
     0.648438}, {0.316, 0.543457}, {0.318, 0.756917}, {0.32, 
    0.683431}, {0.322, 0.697428}, {0.324, 0.781413}, {0.326, 0.79541}, {0.328,
     0.900391}, {0.33, 0.69043}, {0.332, 0.819906}, {0.334, 0.718424}, {0.336,
     0.739421}, {0.338, 0.714925}, {0.34, 0.714925}, {0.342, 
    0.753418}, {0.344, 0.721924}, {0.346, 0.854899}, {0.348, 0.74292}, {0.35, 
    1.03687}, {0.352, 1.01237}, {0.354, 0.942383}, {0.356, 1.07886}, {0.358, 
    0.8479}, {0.36, 0.942383}, {0.362, 1.01587}, {0.364, 0.721924}, {0.366, 
    1.07886}, {0.368, 0.882894}, {0.37, 0.840902}, {0.372, 0.987874}, {0.374, 
    0.970378}, {0.376, 0.924886}, {0.378, 0.910889}, {0.38, 0.8514}, {0.382, 
    0.935384}, {0.384, 1.11385}, {0.386, 1.05786}, {0.388, 1.10685}, {0.39, 
    1.16634}, {0.392, 0.700928}, {0.394, 1.16634}, {0.396, 1.21883}, {0.398, 
    1.12085}, {0.4, 1.05436}, {0.398, 1.31681}, {0.396, 1.12435}, {0.394, 
    1.00887}, {0.392, 1.14884}, {0.39, 1.56527}, {0.388, 0.693929}, {0.386, 
    0.483968}, {0.384, 0.186523}, {0.382, 0.410482}, {0.38, 0.211019}, {0.378,
     0.295003}, {0.376, 0.343994}, {0.374, 
    0.20752}, {0.372, -0.0164388}, {0.37, 0.357992}, {0.368, 
    0.357992}, {0.366, 
    0.0255534}, {0.364, -0.0199382}, {0.362, -0.00594076}, {0.36, 
    0.0605469}, {0.358, -0.212402}, {0.356, -0.0479329}, {0.354, -0.058431}, \
{0.352, -0.289388}, {0.35, -0.0199382}, {0.348, -0.292887}, {0.346, \
-0.191406}, {0.344, -0.310384}, {0.342, -0.166911}, {0.34, -0.348877}, \
{0.338, -0.282389}, {0.336, -0.0444336}, {0.334, -0.380371}, {0.332, \
-0.289388}, {0.33, -0.166911}, {0.328, -0.471354}, {0.326, -0.33488}, {0.324, \
-0.38737}, {0.322, -0.250895}, {0.32, -0.271891}, {0.318, -0.205404}, {0.316, \
-0.446859}, {0.314, -0.390869}, {0.312, -0.299886}, {0.31, -0.33488}, {0.308, \
-0.341878}, {0.306, -0.320882}, {0.304, -0.418864}, {0.302, -0.362874}, {0.3, \
-0.551839}, {0.298, -0.625326}, {0.296, -0.313883}, {0.294, -0.436361}, \
{0.292, -0.436361}, {0.29, -0.247396}, {0.288, -0.341878}, {0.286, \
-0.149414}, {0.284, -0.453857}, {0.282, -0.352376}, {0.28, -0.296387}, \
{0.278, -0.149414}, {0.276, -0.149414}, {0.274, -0.0409342}, {0.272, \
-0.27889}, {0.27, -0.429362}, {0.268, -0.0374349}, {0.266, -0.0829264}, \
{0.264, -0.289388}, {0.262, -0.607829}, {0.26, -0.359375}, {0.258, \
-0.240397}, {0.256, -0.194906}, {0.254, -0.156413}, {0.252, -0.33138}, {0.25, \
-0.488851}, {0.248, -0.320882}, {0.246, -0.292887}, {0.244, -0.394368}, \
{0.242, -0.446859}, {0.24, -0.0619303}, {0.238, -0.0654297}, {0.236, \
-0.114421}, {0.234, -0.509847}, {0.232, -0.103923}, {0.23, -0.152913}, \
{0.228, -0.128418}, {0.226, -0.450358}, {0.224, -0.296387}, {0.222, \
-0.268392}, {0.22, -0.369873}, {0.218, -0.0724284}, {0.216, -0.219401}, \
{0.214, -0.264893}, {0.212, -0.285889}, {0.21, -0.348877}, {0.208, \
-0.485352}, {0.206, -0.219401}, {0.204, -0.0864258}, {0.202, -0.282389}, \
{0.2, -0.366374}, {0.198, -0.443359}, {0.196, -0.180908}, {0.194, -0.159912}, \
{0.192, -0.310384}, {0.19, -0.460856}, {0.188, -0.247396}, {0.186, -0.27889}, \
{0.184, -0.27889}, {0.182, -0.516846}, {0.18, -0.366374}, {0.178, -0.285889}, \
{0.176, -0.296387}, {0.174, -0.166911}, {0.172, -0.264893}, {0.17, \
-0.285889}, {0.168, -0.33488}, {0.166, -0.229899}, {0.164, -0.303385}, \
{0.162, -0.268392}, {0.16, -0.156413}, {0.158, -0.390869}, {0.156, \
-0.341878}, {0.154, -0.159912}, {0.152, -0.397868}, {0.15, -0.236898}, \
{0.148, -0.394368}, {0.146, -0.33138}, {0.144, -0.348877}, {0.142, \
-0.369873}, {0.14, -0.17041}, {0.138, -0.394368}, {0.136, -0.338379}, {0.134, \
-0.268392}, {0.132, -0.306885}, {0.13, -0.2229}, {0.128, -0.275391}, {0.126, \
-0.299886}, {0.124, -0.247396}, {0.122, -0.327881}, {0.12, -0.418864}, \
{0.118, -0.394368}, {0.116, -0.156413}, {0.114, -0.380371}, {0.112, \
-0.352376}, {0.11, -0.135417}, {0.108, -0.397868}, {0.106, -0.404867}, \
{0.104, -0.397868}, {0.102, -0.33138}, {0.1, -0.17391}, {0.098, -0.345378}, \
{0.096, -0.348877}, {0.094, -0.348877}, {0.092, -0.355876}, {0.09, \
-0.366374}, {0.088, -0.436361}, {0.086, -0.359375}, {0.084, -0.292887}, \
{0.082, -0.380371}, {0.08, -0.534342}, {0.078, -0.285889}, {0.076, -0.33138}, \
{0.074, -0.327881}, {0.072, -0.180908}, {0.07, -0.268392}, {0.068, \
-0.310384}, {0.066, -0.352376}, {0.064, -0.436361}, {0.062, -0.229899}, \
{0.06, -0.292887}, {0.058, -0.338379}, {0.056, -0.373372}, {0.054, \
-0.513346}, {0.052, -0.338379}, {0.05, -0.240397}, {0.048, -0.275391}, \
{0.046, -0.513346}, {0.044, -0.446859}, {0.042, -0.338379}, {0.04, \
-0.268392}, {0.038, -0.320882}, {0.036, -0.219401}, {0.034, -0.432861}, \
{0.032, -0.317383}, {0.03, -0.376872}, {0.028, -0.471354}, {0.026, \
-0.345378}, {0.024, -0.366374}, {0.022, -0.401367}, {0.02, -0.527344}, \
{0.018, -0.485352}, {0.016, -0.271891}, {0.014, -0.27889}, {0.012, \
-0.453857}, {0.01, -0.527344}, {0.008, -0.366374}, {0.006, -0.366374}, \
{0.004, -0.194906}, {0.002, -0.369873}, {1.25*10^-16, -0.485352}, {-0.002, \
-0.597331}, {-0.004, -0.338379}, {-0.006, -0.380371}, {-0.008, -0.394368}, \
{-0.01, -0.443359}, {-0.012, -0.285889}, {-0.014, -0.436361}, {-0.016, \
-0.516846}, {-0.018, -0.418864}, {-0.02, -0.338379}, {-0.022, -0.320882}, \
{-0.024, -0.317383}, {-0.026, -0.33488}, {-0.028, -0.327881}, {-0.03, \
-0.2264}, {-0.032, -0.317383}, {-0.034, -0.359375}, {-0.036, -0.38387}, \
{-0.038, -0.576335}, {-0.04, -0.677816}, {-0.042, -0.418864}, {-0.044, \
-0.558838}, {-0.046, -0.450358}, {-0.048, -0.499349}, {-0.05, -0.352376}, \
{-0.052, -0.527344}, {-0.054, -0.159912}, {-0.056, -0.523844}, {-0.058, \
-0.257894}, {-0.06, -0.460856}, {-0.062, -0.369873}, {-0.064, -0.397868}, \
{-0.066, -0.614827}, {-0.068, -0.516846}, {-0.07, -0.257894}, {-0.072, \
-0.558838}, {-0.074, -0.415365}, {-0.076, -0.730306}, {-0.078, -0.296387}, \
{-0.08, -0.523844}, {-0.082, -0.49585}, {-0.084, -0.502848}, {-0.086, \
-0.562337}, {-0.088, -0.562337}, {-0.09, -0.310384}, {-0.092, -0.373372}, \
{-0.094, -0.373372}, {-0.096, -0.320882}, {-0.098, -0.380371}, {-0.1, \
-0.530843}, {-0.102, -0.397868}, {-0.104, -0.572835}, {-0.106, -0.516846}, \
{-0.108, -0.408366}, {-0.11, -0.380371}, {-0.112, -0.527344}, {-0.114, \
-0.478353}, {-0.116, -0.579834}, {-0.118, -0.425863}, {-0.12, -0.310384}, \
{-0.122, -0.457357}, {-0.124, -0.520345}, {-0.126, -0.338379}, {-0.128, \
-0.27889}, {-0.13, -0.579834}, {-0.132, -0.555339}, {-0.134, -0.593831}, \
{-0.136, -0.541341}, {-0.138, -0.464355}, {-0.14, -0.534342}, {-0.142, \
-0.450358}, {-0.144, -0.432861}, {-0.146, -0.366374}, {-0.148, -0.478353}, \
{-0.15, -0.443359}, {-0.152, -0.488851}, {-0.154, -0.681315}, {-0.156, \
-0.635824}, {-0.158, -0.345378}, {-0.16, -0.49585}, {-0.162, -0.597331}, \
{-0.164, -0.502848}, {-0.166, -0.485352}, {-0.168, -0.621826}, {-0.17, \
-0.516846}, {-0.172, -0.625326}, {-0.174, -0.572835}, {-0.176, -0.478353}, \
{-0.178, -0.569336}, {-0.18, -0.569336}, {-0.182, -0.569336}, {-0.184, \
-0.712809}, {-0.186, -0.723307}, {-0.188, -0.569336}, {-0.19, -0.586833}, \
{-0.192, -0.604329}, {-0.194, -0.674316}, {-0.196, -0.688314}, {-0.198, \
-0.614827}, {-0.2, -0.747803}};

Splitting the data

data2 = {First[#], Flatten[Rest[#], 1]} &@Split[data, #1[[1]] < #2[[1]] &];

f1[x_] = FindFormula[data2[[1]], x,
  PerformanceGoal -> "Quality",
  SpecificityGoal -> "High",
  RandomSeeding -> 0]

(* 0.298829 + 0.356977 x - 3.21678 x^2 + 18.5331 x^3 *)

x1min = Min[data2[[1, All, 1]]]

(* -0.198 *)

f2[x_] = FindFormula[data2[[2]], x,
  PerformanceGoal -> "Quality",
  SpecificityGoal -> "High",
  RandomSeeding -> 0]

(* -0.377387 + 0.767637 x - 6.16336 x^2 - 13.9907 x^3 + 429.278 x^4 - 
 111.08 x^5 - 8809.58 x^6 + 18052.7 x^7 *)

x2min = Min[data2[[2, All, 1]]]

(* -0.2 *)

While the starting point for each curve (x1min and x2min) is taken from the respective data segments, the stopping point for both curves is where the curves intersect, i.e., where f1[x] == f2[x]

xmax = x /. NSolve[f1[x] == f2[x], x, Reals][[1]]

(* 0.394405 *)

Show[
 ListLinePlot[Flatten[data2, 1],
  PlotStyle -> LightBlue],
 Plot[f1[x], {x, x1min, xmax}, PlotStyle -> Red, PlotRange -> All],
 Plot[f2[x], {x, x2min, xmax}, PlotStyle -> Red, PlotRange -> All],
 PlotRange -> {{-0.3, 0.5}, {-1, 2}}]

enter image description here

$\endgroup$
  • $\begingroup$ Thank you Bob this method is the best for my case. I will try it for the other curves I have and let you know the results. :) $\endgroup$ – Mat-laut Jul 11 '18 at 6:34
  • $\begingroup$ I would appreciate if you can tell me this line is for what in your code: xmax = x /. NSolve[f1[x] == f2[x], x, Reals][[1]] $\endgroup$ – Mat-laut Jul 11 '18 at 8:22
  • $\begingroup$ @HendMkaouar - It identifies the max value of x to use in drawing each curve, i.e., stop when the two curves intersect. $\endgroup$ – Bob Hanlon Jul 11 '18 at 15:12
8
$\begingroup$

(Comment in order to clarify the question.)

Is this what you want to get?

enter image description here

If yes, that can be done with Quantile Regression as explained in this answer for the question you linked in your answer, "Remove noise from data".

Or using MovingAverage[data,20] as @kirma commented.

enter image description here

$\endgroup$
  • $\begingroup$ yes this is good enough but how did you choose the parameters: 0.25 and 0.75? $\endgroup$ – Mat-laut Jul 10 '18 at 13:22
  • $\begingroup$ @HendMkaouar Well, I just assumed and then verified that your data has equal parts above and below the quantile 0.5. Since you expressed interest in this solution I post better explanations in my answer. $\endgroup$ – Anton Antonov Jul 10 '18 at 13:25
  • $\begingroup$ @HendMkaouar Also, please consider editing your question with clarifications. (It would get more upvotes...) $\endgroup$ – Anton Antonov Jul 10 '18 at 13:31
  • $\begingroup$ Thank you for the reply. I will try to clarify more my question with a picture that explains my data and the data it should be. $\endgroup$ – Mat-laut Jul 10 '18 at 13:34
4
$\begingroup$

Another way might be to use EstimatedBackground to estimate the data's envelope and then average it.

First split the data into two:

{data1, data2} = {#1, Join[##2]} & @@ Split[data, Sign[#1[[1]] - #2[[1]]] == -1 &];

The backgrounds of the first half of the data and its average:

lo1 = Transpose[{#1, EstimatedBackground[#2, 10]}] & @@ Transpose[data1];
hi1 = Transpose[{#1, -EstimatedBackground[-#2, 10]}] & @@ Transpose[data1];
smooth1 = Mean[{lo1, hi1}];

ListLinePlot[{data1, lo1, hi1, smooth1}, PlotStyle -> {Opacity[.5], Red, Red, Black}]

enter image description here

Both pieces of data:

lo2 = Transpose[{#1, EstimatedBackground[#2]}] & @@ Transpose[data2];
hi2 = Transpose[{#1, -EstimatedBackground[-#2]}] & @@ Transpose[data2];
smooth2 = Mean[{lo2, hi2}];

Show[
  ListLinePlot[{data1, smooth1}, PlotStyle -> {Opacity[.5], Red}],
  ListLinePlot[{data2, smooth2}, PlotStyle -> {Opacity[.5], Red}, PlotRange -> All],
  PlotRange -> All
]

enter image description here

$\endgroup$
  • $\begingroup$ Thank you Chip for the reply, this one is also good :) $\endgroup$ – Mat-laut Jul 11 '18 at 6:35
4
$\begingroup$

I'm surprised no one used actual signal filtering yet.

fq = 0.015;
ts = Table[{i, data[[i, 2]]}, {i, 1, Length[data]}];
fx = LowpassFilter[TimeSeries@ts, Quantity[fq, "Hertz"]];
ts2 = Table[{i, data[[i, 1]]}, {i, 1, Length[data]}];
fx2 = LowpassFilter[TimeSeries@ts2, Quantity[fq, "Hertz"]];
ListPlot[Table[{fx2[i], fx[i]}, {i, 1, Length[data]}]]

fq=0.015:

enter image description here

fq=0.005:

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.