6
$\begingroup$

Suppose I have an unknown function $f(x)$ that is monotonic, convex and nonlinear by definition. I somehow evaluated it at 1000 nonuniform $x$ points. By ploting these data, I found a very obvious nonlinear curve with a dozen of outliers, which, e.g., contain sudden small or large peaks and valleys and violate monotonicity and convexity. How to carry out outlier recognition automatically? Sth. more systematic than inspecting by eyes...

Note that my ultimate purpose is not finding any fit of these data and I don't want any modification among the data except excluding outliers.

The data set is (updated, more reasonable, different from the one plotted in the comment below, apology for that I cannot retain the original one due to character limit)

data = {0,197.1278166},{3.356150145 10^-5,197.1004078},{6.745694819 10^-5,197.0729913},{0.0001016863402,197.045061},{0.0001362496775,197.01686},{0.0001711469601,196.9884324},{0.000206378188,196.9597409},{0.0002419433612,196.9307115},{0.0002778424797,196.9014819},{0.0003140755434,196.872016},{0.0003506425525,196.8422888},{0.0003875435068,196.8123176},{0.0004247784064,196.7820562},{0.0004623472513,196.7515632},{0.0005002500415,196.7208153},{0.000538486777,196.6896979},{0.0005770574578,196.6584585},{0.0006159620839,196.6269528},{0.0006552006552,196.5952411},{0.0006947731718,196.5632462},{0.0007346796338,196.5309877},{0.000774920041,196.4984332},{0.0008154943935,196.4656562},{0.0008564026913,196.4326006},{0.0008976449343,196.3992963},{0.0009392211227,196.365732},{0.0009811312564,196.3319839},{0.001023375335,196.2979535},{0.00106595336,196.2636788},{0.001108865329,196.2290179},{0.001152111244,196.1942448},{0.001195691104,196.159195},{0.001239604909,196.1239301},{0.00128385266,196.0884187},{0.001328434356,196.0526233},{0.001373349997,196.0165869},{0.001418599584,195.9803034},{0.001464183116,195.9438184},{0.001510100593,195.9069059},{0.001556352015,195.8698333},{0.001602937383,195.8325826},{0.001649856696,195.7950497},{0.001697109954,195.7573184},{0.001744697158,195.7193407},{0.001792618306,195.6811246},{0.001840873401,195.6426487},{0.00188946244,195.6039494},{0.001938385425,195.5650393},{0.001987642355,195.5259065},{0.00203723323,195.4864771},{0.00208715805,195.4468092},{0.002137416816,195.4069093},{0.002188009527,195.3667779},{0.002238936184,195.3263106},{0.002290196786,195.2857357},{0.002341791333,195.2448866},{0.002393719825,195.2037633},{0.002445982262,195.1624498},{0.002498578645,195.1209472},{0.002551508974,195.0791495},{0.002604773247,195.0372058},{0.002658371466,194.9949096},{0.00271230363,194.9524911},{0.002766569739,194.9098417},{0.002821169794,194.8672176},{0.002876103794,194.8240065},{0.002931371739,194.7805768},{0.002986973629,194.736972},{0.003042909465,194.702772},{0.003099179246,194.6586471},{0.003155782972,194.6141539},{0.003212720644,194.5696212},{0.003269992261,194.5248077},{0.003327597823,194.5436408},{0.00338553733,194.4342508},{0.003443810783,194.3887827},{0.003502418181,194.3432184},{0.003561359525,194.2972514},{0.003620634813,194.2511},{0.003680244047,194.2047559},{0.003740187226,194.1581676},{0.003800464351,194.1113705},{0.003861075421,194.064177},{0.003922020436,194.0169291},{0.003983299396,193.9695131},{0.004044912302,193.9217952},{0.004106859153,193.8738824},{0.004169139949,193.8257357},{0.00423175469,193.7773793},{0.004294703377,193.7288066},{0.004357986009,193.6800275},{0.004421602587,193.6307837},{0.004485553109,193.5822356},{0.004549837577,193.5328221},{0.004614455991,193.4831627},{0.004679408349,193.4332837},{0.004744694653,193.3832249},{0.004810314902,193.3329423},{0.004876269096,193.2824868},{0.004942557236,193.2308879},{0.005009179321,193.1799623},{0.005076135351,193.1282019},{0.005143425327,193.076852},{0.005211049248,193.0252946},{0.005279007114,192.9735373},{0.005347298925,192.9215745},{0.005415924682,192.8693928},{0.005484884384,192.8170039},{0.005554178031,192.7644067},{0.005623805624,192.7116614},{0.005693767162,192.6632774},{0.005764062645,192.610056},{0.005834692073,192.5566516},{0.005905655447,192.4989477},{0.005976952766,192.4451613},{0.00604858403,192.3911403},{0.00612054924,192.3368714},{0.006192848395,192.2823985},{0.006265481495,192.2279216},{0.00633844854,192.1732041},{0.006411749531,192.1188254},{0.006485384467,192.0635985},{0.006559353348,192.00814},{0.006633656175,191.9525783},{0.006708292947,191.896843},{0.006783263664,191.8408394},{0.006858568326,191.7846399},{0.006934206934,191.7282461},{0.007010179487,191.6716881},{0.007086485986,191.6149185},{0.007163126429,191.5579129},{0.007240100818,191.5007167},{0.007317409152,191.4433843},{0.007395051432,191.3858193},{0.007473027657,191.3280679},{0.007551337827,191.2700583},{0.007629981942,191.2117456},{0.007708960003,191.1534528},{0.007788272008,191.0949794},{0.00786791796,191.0363179},{0.007947897856,190.9769944},{0.008028211698,190.9179446},{0.008108859485,190.8586478},{0.008189841217,190.7992471},{0.008271156895,190.7396372},{0.008352806518,190.679838},{0.008434790086,190.619893},{0.0085171076,190.5597393},{0.008599759058,190.4999113},{0.008682744463,190.4394211},{0.008766063812,190.3788123},{0.008849717107,190.3179419},{0.008933704347,190.2569127},{0.009018025532,190.195623},{0.009102680662,190.134266},{0.009187669738,190.072727},{0.009272992759,190.0109841},{0.009358649726,189.9494614},{0.009444640637,189.8873756},{0.009530965494,189.8256654},{0.009617624297,189.7633164},{0.009704617044,189.7007075},{0.009791943737,189.637898},{0.009879604375,189.5750092},{0.009967598958,189.5118553},{0.01005592749,189.4485701},{0.01014458996,189.3851623},{0.01023358638,189.3216322},{0.01032291674,189.2578523},{0.01041258105,189.1938933},{0.01050257931,189.1297504},{0.01059291151,189.0654914},{0.01068357766,189.0011197},{0.01077457775,188.9365838},{0.01086591178,188.8719057},{0.01095757976,188.8069117},{0.01104958169,188.7418705},{0.01114191756,188.6767146},{0.01123458738,188.611512},{0.01132759114,188.5459706},{0.01142092885,188.4802746},{0.01151460051,188.4144563},{0.0116086061,188.3485018},{0.01170294565,188.2823204},{0.01179761914,188.2162067},{0.01189262657,188.1496851},{0.01198796795,188.0830608},{0.01208364328,188.0047476},{0.01217965255,187.9378282},{0.01227599576,187.8707175},{0.01237267292,187.803437},{0.01246968403,187.7360423},{0.01256702908,187.6684166},{0.01266470808,187.6005729},{0.01276272102,187.5327545},{0.01286106791,187.4647985},{0.01295974874,187.3966443},{0.01305876352,187.3403829},{0.01315811224,187.2717695},{0.01325779491,187.203258},{0.01335781152,187.1345019},{0.01345816208,187.0656647},{0.01355884659,186.9967007},{0.01365986504,186.9275895},{0.01376121743,186.8583423},{0.01386290377,186.788935},{0.01396492406,186.7193894},{0.01406727829,186.6497106},{0.01416996646,186.5798815},{0.01427298858,186.5101529},{0.01437634465,186.4403075},{0.01448003466,186.370164},{0.01458405862,186.2999133},{0.01468841652,186.2294668},{0.01479310837,186.1588527},{0.01489813416,186.0878101},{0.0150034939,186.0169655},{0.01510918759,185.9459311},{0.01521521522,185.8746949},{0.01532157679,185.8033647},{0.01542827231,185.7319244},{0.01553530177,185.6604904},{0.01564266518,185.5887671},{0.01575036254,185.5170783},{0.01585839384,185.4451875},{0.01596675909,185.3732202},{0.01607545828,185.3010854},{0.01618449141,185.2288148},{0.0162938585,185.1565485},{0.01640355952,185.0840694},{0.0165135945,185.011647},{0.01662396341,184.9387974},{0.01673466628,184.8659304},{0.01684570308,184.7929044},{0.01695707384,184.7198332},{0.01706877854,184.6465929},{0.01718081718,184.5733122},{0.01729318977,184.4997967},{0.0174058963,184.4276583},{0.01751893678,184.3540281},{0.01763231121,184.280298},{0.01774601958,184.206201},{0.0178600619,184.1322271},{0.01797443816,184.0580777},{0.01808914836,183.9837931},{0.01820419252,183.9094314},{0.01831957061,183.8349641},{0.01843528266,183.7603299},{0.01855132864,183.6855938},{0.01866770858,183.6125457},{0.01878442245,183.5380365},{0.01890147028,183.4631218},{0.01901885205,183.388047},{0.01913656776,183.3114483},{0.01925461742,183.2364953},{0.01937300102,183.161083},{0.01949171857,183.0855589},{0.01961077007,183.0102167},{0.01973015551,182.9345003},{0.0198498749,182.9124951},{0.01996992823,182.836446},{0.0200903155,182.747303},{0.02021103672,182.6307561},{0.02033209189,182.5551697},{0.020453481,182.4781509},{0.02057520406,182.3995031},{0.02069726106,182.3231911},{0.02081965201,182.2468834},{0.02094237691,182.1702375},{0.02106543574,182.0935114},{0.02118882853,182.0166714},{0.02131255526,181.9397494},{0.02143661593,181.8627249},{0.02156101055,181.7856132},{0.02168573912,181.7084094},{0.02181080163,181.6311061},{0.02193619808,181.5536941},{0.02206192848,181.4763405},{0.02218799283,181.3987658},{0.02231439112,181.3210121},{0.02244112336,181.2433051},{0.02256818954,181.1619992},{0.02269558967,181.0841404},{0.02282332374,181.0061894},{0.02295139176,180.9281675},{0.02307979372,180.8500109},{0.02320852963,180.7718139},{0.02333759948,180.6936051},{0.02346700328,180.6169435},{0.02359674103,180.5386278},{0.02372681272,180.4612075},{0.02385721835,180.3827548},{0.02398795793,180.3042083},{0.02411903146,180.2255088},{0.02425043893,180.146702},{0.02438218035,180.0681063},{0.02451425571,179.9889092},{0.02464666501,179.9125043},{0.02477940827,179.8334365},{0.02491248546,179.7530359},{0.02504589661,179.6771424},{0.02517964169,179.5978507},{0.02531372073,179.5184052},{0.02544813371,179.4937656},{0.02558288063,179.4134635},{0.0257179615,179.3332739},{0.02585337631,179.1990209},{0.02598912507,179.1193987},{0.02612520778,179.0363637},{0.02626162443,178.9565442},{0.02639837502,178.8766865},{0.02653545956,178.7967004},{0.02667287805,178.7178436},{0.02681063048,178.6377664},{0.02694871686,178.5576216},{0.02708713718,178.4775176},{0.02722589145,178.3972777},{0.02736497966,178.3170332},{0.02750440182,178.2367385},{0.02764415792,178.1562691},{0.02778424797,178.0757643},{0.02792467196,177.9952589},{0.0280654299,177.9148052},{0.02820652178,177.8341355},{0.02834794761,177.7528213},{0.02848970739,177.6716878},{0.02863180111,177.5909004},{0.02877422877,177.5098097},{0.02891699038,177.4289368},{0.02906008594,177.3479483},{0.02920351544,177.2669914},{0.02934727889,177.182478},{0.02949137628,177.1013995},{0.02963580762,177.0203868},{0.0297805729,176.9392446},{0.02992567213,176.8580698},{0.0300711053,176.7767435},{0.03021687242,176.6962045},{0.03036297348,176.6149249},{0.03050940849,176.5335917},{0.03065617745,176.452031},{0.03080328034,176.3705825},{0.03095071719,176.2891502},{0.03109848798,176.2531801},{0.03124659271,176.1289193},{0.0313950314,176.0472398},{0.03154380402,175.965686},{0.03169291059,175.8840234},{0.03184235111,175.8023073},{0.03199212557,175.7204286},{0.03214223398,175.6395877},{0.03229267633,175.5577963},{0.03244345263,175.4760437},{0.03259456287,175.3942196},{0.03274600706,175.376237},{0.03289778519,175.2296923},{0.03304989727,175.1509588},{0.03320234329,175.0689788},{0.03335512326,174.98656},{0.03350823718,174.906809},{0.03366168504,174.8214931},{0.03381546684,174.739502},{0.03396958259,174.6573538},{0.03412403229,174.5753838},{0.03427881593,174.4931468},{0.03443393352,174.4109092},{0.03458938505,174.3286671},{0.03474517052,174.2450544},{0.03490128995,174.1627534},{0.03505774331,174.0804378},{0.03521453063,173.9981073},{0.03537165189,173.9165271},{0.03552910709,173.8341437},{0.03568689624,173.7504133},{0.03584501933,173.6658442},{0.03600347637,173.5833748},{0.03616226735,173.5023612},{0.03632139228,173.4199275},{0.03648085116,173.3373531},{0.03664064398,173.2548041},{0.03680077075,173.1723048},{0.03696123146,173.0899019},{0.03712202611,173.0073416},{0.03728315471,172.9249754},{0.03744461726,172.8415334},{0.03760641375,172.7593304},{0.03776854419,172.6767716},{0.03793100857,172.595349},{0.0380938069,172.5196201},{0.03825693917,172.412622},{0.03842040539,172.3300209},{0.03858420556,172.2394988},{0.03874833967,172.1567139},{0.03891280772,172.0741175},{0.03907760972,172.0055043},{0.03924274566,171.9091177},{0.03940821556,171.8513236},{0.03957401939,171.7687034},{0.03974015717,171.6860516},{0.0399066289,171.6034113},{0.04007343457,171.5203097},{0.04024057419,171.4378736},{0.04040804775,171.3552247},{0.04057585525,171.2734368},{0.04074399671,171.1907934},{0.04091247211,171.108151},{0.04108128145,171.0259058},{0.04125042474,170.9432347},{0.04141990197,170.8566748},{0.04158971315,170.7695756},{0.04175985827,170.6870868},{0.04193033734,170.6044522},{0.04210115036,170.5219059},{0.04227229732,170.4272953},{0.04244377822,170.3446737},{0.04261559307,170.1942995},{0.04278774187,170.1117729},{0.04296022461,170.1127368},{0.0431330413,170.0303557},{0.04330619193,169.9477709},{0.04347967651,169.8652415},{0.04365349503,169.7827082},{0.0438276475,169.7008097},{0.04400213391,169.6183312},{0.04417695427,169.5359436},{0.04435210857,169.4536046},{0.04452759682,169.3620933},{0.04470341902,169.280066},{0.04487957515,169.1970223},{0.04505606524,169.114611},{0.04523288927,169.032219},{0.04541004724,168.9498754},{0.04558753917,168.8674773},{0.04576536503,168.7845111},{0.04594352484,168.7031782},{0.0461220186,168.6207711},{0.0463008463,168.5384551},{0.04648000795,168.45621},{0.04665950354,168.3739174},{0.04683933308,168.2916501},{0.04701949656,168.2115635},{0.04719999399,168.1293878},{0.04738082536,168.0458698},{0.04756199068,166.6727933},{0.04774348995,167.5950992},{0.04792532315,167.7994518},{0.04810749031,167.715341},{0.04828999141,167.6333352},{0.04847282645,167.5513337},{0.04865599544,167.469929},{0.04883949838,167.3880376},{0.04902333526,167.3103549},{0.04920750609,167.2280614},{0.04939201086,167.1463252},{0.04957684958,167.0638852},{0.04976202224,166.9821337},{0.04994752885,166.9005617},{0.0501333694,166.8188312},{0.0503195439,166.7370606},{0.05050605234,166.6543451},{0.05069289473,166.5727542},{0.05088007106,166.4911428},{0.05106758134,166.4097354},{0.05125542557,166.3284302},{0.05144360374,166.2559003},{0.05163211585,166.1737082},{0.05182096191,166.092188},{0.05201014192,166.0108887},{0.05219965587,165.9296466},{0.05238950377,165.8482824},{0.05257968561,165.7666369},{0.05277020139,165.6852117},{0.05296105113,165.6040816},{0.0531522348,165.5229809},{0.05334375243,165.441487},{0.05353560399,165.3603353},{0.05372778951,165.2793307},{0.05392030897,165.1604849},{0.05411316237,165.0796811},{0.05430634972,165.0383012},{0.05449987101,164.9542571},{0.05469372625,164.8764578},{0.05488791544,164.7841347},{0.05508243857,164.7033381},{0.05527729564,164.6222814},{0.05547248667,164.5427669},{0.05566801163,164.4622751},{0.05586387054,164.3818476},{0.0560600634,164.3022186},{0.0562565902,164.2218004},{0.05645345095,164.14146},{0.05665064564,164.0612435},{0.05684817428,163.9805505},{0.05704603686,163.9007545},{0.05724423339,163.8204748},{0.05744276386,163.7408617},{0.05764162828,163.6607542},{0.05784082665,163.5807716},{0.05804035896,163.5015196},{0.05824022521,163.4217131},{0.05844042541,163.3414346},{0.05864095956,163.2616512},{0.05884182765,163.1819282},{0.05904302969,163.1025908},{0.05924456567,163.0348497},{0.05944643559,162.9553867},{0.05964863947,162.875767},{0.05985117728,162.7964336},{0.06005404904,162.7169686},{0.06025725475,162.6374781},{0.06046079441,162.5584071},{0.060664668,162.4793074},{0.06086887555,162.4003465},{0.06107341704,162.3213533},{0.06127829247,162.242015},{0.06148350185,162.1633587},{0.06168904518,162.0844599},{0.06189492245,162.0047134},{0.06210113366,161.9264259},{0.06230767882,161.8477088},{0.06251455793,161.7691371},{0.06272177098,161.7454818},{0.06292931798,161.6125648},{0.06313719892,161.5339954},{0.0633454138,161.4556998},{0.06355396264,161.3775162},{0.06376284541,161.299579},{0.06397206214,161.2216921},{0.06418161281,161.1389801},{0.06439149742,161.0612204},{0.06460171598,160.9838439},{0.06481226848,160.9059137},{0.06502315493,160.8280933},{0.06523437533,160.7506325},{0.06544592967,160.6728376},{0.06565781795,160.5953233},{0.06587004018,160.5161053},{0.06608259636,160.4356336},{0.06629548648,160.3584049},{0.06650871055,160.2808348},{0.06672226856,160.2038882},{0.06693616051,160.1268593},{0.06715038642,160.0498597},{0.06736494626,159.9803767},{0.06757984006,159.9037836},{0.0677950678,159.8270333},{0.06801062948,159.7504824},{0.06822652511,159.6738535},{0.06844275468,159.5972941},{0.0686593182,159.518468},{0.06887621567,159.4417531},{0.06909344708,159.3605843},{0.06931101243,159.2844222},{0.06952891173,159.2063131},{0.06974714498,159.1254675},{0.06996571217,159.0492498},{0.0701846133,158.973343},{0.07040384839,158.8977789},{0.07062341741,158.822092},{0.07084332038,158.7472791},{0.0710635573,158.6717238},{0.07128412816,158.6033784},{0.07150503297,158.5280712},{0.07172627173,158.4565419},{0.07194784442,158.3813409},{0.07216975107,158.306035},{0.07239199166,158.2307326},{0.07261456619,158.1557767},{0.07283747467,158.0806633},{0.0730607171,158.0384986},{0.07328429347,157.9253107},{0.07350820378,157.850755},{0.07373244804,157.776082},{0.07395702625,157.7082962},{0.0741819384,157.6341445},{0.0744071845,157.5655},{0.07463276454,157.491307},{0.07485867853,157.4173646},{0.07508492646,157.3434915},{0.07531150834,157.2675217},{0.07553842416,157.1937339},{0.07576567393,157.1200935},{0.07599325764,157.0415573},{0.0762211753,156.9681381},{0.07644942691,156.898088},{0.07667801246,156.8247235},{0.07690693195,156.7514581},{0.07713618539,156.6792799},{0.07736577278,156.6077228},{0.07759569411,156.5352384},{0.07782594939,156.4631066},{0.07805653861,156.3968898},{0.07828746177,156.322394},{0.07851871889,156.249908},{0.07875030994,156.1772819},{0.07898223495,156.1048976},{0.07921449389,156.0300454},{0.07944708679,155.9576447},{0.07968001362,155.8855744},{0.07991327441,155.8136102},{0.08014686914,155.7414248},{0.08038079781,155.669621},{0.08061506043,155.5980189},{0.080849657,155.5265474},{0.08108458751,155.4551591},{0.08131985196,155.3837723},{0.08155545036,155.3114967},{0.08179138271,155.2403093},{0.082027649,155.1713536},{0.08226424924,155.1005714},{0.08250118342,155.0298306},{0.08273845155,154.9581803},{0.08297605362,154.8875969},{0.08321398964,154.8171418},{0.0834522596,154.7467593},{0.08369086351,154.6765923},{0.08392980136,154.60642},{0.08416907316,154.5363371},{0.0844086789,154.4663844},{0.08464861859,154.3965552},{0.08488889223,154.327151},{0.08512949981,154.2575166},{0.08537044133,154.1879296},{0.0856117168,154.1185324},{0.08585332622,154.0490914},{0.08609526958,153.9800443},{0.08633754689,153.9108881},{0.08658015814,153.8419361},{0.08682310334,153.7721854},{0.08706638248,153.7033862},{0.08730999557,153.6347458},{0.0875539426,153.564328},{0.08779822358,153.4965151},{0.0880428385,153.428174},{0.08828778737,153.3600168},{0.08853307018,153.2919627},{0.08877868694,153.2240426},{0.08902463765,153.1565331},{0.0892709223,153.0866829},{0.08951754089,153.0188238},{0.08976449343,152.9516902},{0.09001177992,152.8843626},{0.09025940035,152.8171265},{0.09050735473,152.7498156},{0.09075564305,152.6826869},{0.09100426532,152.6158369},{0.09125322153,152.5490504},{0.09150251169,152.4847273},{0.09175213579,152.4182175},{0.09200209384,152.3517801},{0.09225238583,152.2852707},{0.09250301177,152.219286},{0.09275397165,152.1532693},{0.09300526548,152.0873741},{0.09325689326,152.0216154},{0.09350885498,151.9498943},{0.09376115064,151.8844382},{0.09401378025,151.8189028},{0.09426674381,151.7536375},{0.09452004131,151.6885716},{0.09477367276,150.7643714},{0.09502763815,151.5585309},{0.09528193748,151.4955343},{0.09553657077,151.4309819},{0.09579153799,151.3664682},{0.09604683917,151.3020992},{0.09630247428,151.2375802},{0.09655844335,151.1790687},{0.09681474636,151.0955346},{0.09707138331,151.0510999},{0.09732835421,150.9874088},{0.09758565905,150.9236689},{0.09784329784,150.896193},{0.09810127058,150.8325334},{0.09835957726,150.7336542},{0.09861821788,150.6579914},{0.09887719246,150.595031},{0.09913650097,150.5322366},{0.09939614343,150.4694859},{0.09965611984,150.4069638},{0.09991643019,150.3444604},{0.1001770745,150.2821349},{0.1004380527,150.2198833},{0.1006993649,150.1682288},{0.1009610111,150.1062845},{0.1012229911,150.0445273},{0.1014853052,149.9847908},{0.1017479531,149.9222688},{0.102010935,149.8608381},{0.1022742509,149.7995485},{0.1025379007,149.738548},{0.1028018845,149.6774235},{0.1030662021,149.6166871},{0.1033308538,149.5559309},{0.1035958394,149.4954904},{0.1038611589,149.4353288},{0.1041268124,149.3750501},{0.1043927998,149.314987},{0.1046591212,149.2550035},{0.1049257765,149.1951236},{0.1051927657,149.1349349},{0.1054600889,149.0753183},{0.1057277461,149.0158887},{0.1059957372,148.956563},{0.1062640622,148.8973945},{0.1065327212,148.8383551},{0.1068017141,148.7794661},{0.107071041,148.721649},{0.1073407018,148.6627816},{0.1076106966,148.6043257},{0.1078810253,148.5461982},{0.108151688,148.4879718},{0.1084226846,148.4244003},{0.1086940151,148.3719278},{0.1089656796,148.3141683},{0.109237678,148.2565288},{0.1095100104,148.2002034},{0.1097826768,148.1429433},{0.110055677,148.0857339},{0.1103290112,148.028453},{0.1106026794,147.9715658},{0.1108766815,147.914738},{0.1111510176,147.8582529},{0.1114256876,147.8017829},{0.1117006915,147.7452773},{0.1119760294,147.6889778},{0.1122517012,147.6324342},{0.112527707,147.5687485},{0.1128040467,147.5132086},{0.1130807204,147.4575143},{0.113357728,147.3990253},{0.1136350696,147.3517156},{0.1139127451,147.297189},{0.1141907546,147.2412294},{0.114469098,147.186515},{0.1147477753,147.1320073},{0.1150267866,147.0775351},{0.1153061318,147.0232097},{0.115585811,146.9678205},{0.1158658241,146.913821},{0.1161461712,146.8599706},{0.1164268522,146.8062321},{0.1167078672,146.7526677},{0.1169892161,146.6990783},{0.1172708989,146.6459931},{0.1175529157,146.592912},{0.1178352665,146.5399935},{0.1181179511,146.4872617},{0.1184009698,146.4371685},{0.1186843224,146.3858241},{0.1189680089,146.3334888},{0.1192520293,146.2813497},{0.1195363838,146.2279793},{0.1198210721,146.1766692},{0.1201060944,146.1249808},{0.1203914507,146.0764323},{0.1206771409,146.0247479},{0.120963165,145.9742258},{0.1212495231,145.9220015},{0.1215362151,145.871107},{0.1218232411,145.8196001},{0.122110601,145.7689301},{0.1223982949,145.7184674},{0.1226863227,145.6681545},{0.1229746844,145.6180323},{0.1232633801,145.5682446},{0.1235524098,145.5184573},{0.1238417734,145.4688998},{0.1241314709,145.4195142},{0.1244215024,145.370311},{0.1247118678,145.321216},{0.1250025672,145.2722694},{0.1252936005,145.2233876},{0.1255849678,145.1749977},{0.125876669,145.1266203},{0.1261687042,145.0784784},{0.1264610733,145.029315},{0.1267537763,144.9815222},{0.1270468133,144.933133},{0.1273401842,144.8854934},{0.1276338891,144.8380634},{0.1279279279,144.7908911},{0.1282223007,144.7443241},{0.1285170074,144.6974303},{0.1288120481,144.6372317},{0.1291074227,144.6054064},{0.1294031312,144.5560988},{0.1296991737,144.5081254},{0.1299955502,144.4621071},{0.1302922606,144.4165009},{0.1305893049,144.3707081},{0.1308866832,144.3251677},{0.1311843954,144.2820557},{0.1314824416,144.2368588},{0.1317808217,144.1919367},{0.1320795357,144.1471125},{0.1323785838,144.1024277},{0.1326779657,144.0582624},{0.1329776816,144.0136203},{0.1332777314,143.9683443},{0.1335781152,143.9244033},{0.133878833,143.8804986},{0.1341798846,143.8369465},{0.1344812703,143.7951425},{0.1347829898,143.7518746},{0.1350850433,143.7088609},{0.1353874308,143.6656186},{0.1356901522,143.6228768},{0.1359932076,143.5798892},{0.1362965968,143.5375741},{0.1366003201,143.4953213},{0.1369043773,143.4534346},{0.1372087684,143.4115725},{0.1375134935,143.3699146},{0.1378185525,143.3284516},{0.1381239455,143.2871888},{0.1384296724,143.2463075},{0.1387357332,143.2054027},{0.139042128,143.1638454},{0.1393488568,143.1230135},{0.1396559195,143.0802327},{0.1399633161,143.0396385},{0.1402710467,142.9991716},{0.1405791112,142.9600239},{0.1408875097,142.9200503},{0.1411962421,142.8831554},{0.1415053085,142.8447254},{0.1418147088,142.8049761},{0.142124443,142.7665621},{0.1424345112,142.7277853},{0.1427449134,142.6904091},{0.1430556495,142.6520269},{0.1433667195,142.6139525},{0.1436781235,142.5760352},{0.1439898614,142.5376418},{0.1443019333,142.5000421},{0.1446143391,142.4626559},{0.1449270789,142.4251467},{0.1452401526,142.3881464},{0.1455535602,142.3512217},{0.1458673018,142.3144979},{0.1461813774,142.27878},{0.1464957869,142.242528},{0.1468105303,142.2065052},{0.1471256077,142.1705542},{0.147441019,142.1348204},{0.1477567643,142.0993145},{0.1480728435,142.0626915},{0.1483892566,141.9901571},{0.1487060038,141.9934823},{0.1490230848,141.9587522},{0.1493404998,141.9260329},{0.1496582487,141.8637437},{0.1499763316,141.855457},{0.1502947485,141.8208833},{0.1506134992,141.7871529},{0.150932584,141.7536983},{0.1512520026,141.7202958},{0.1515717552,141.6488299},{0.1518918418,141.6552988},{0.1522122623,141.62251},{0.1525330168,141.5900088},{0.1528541051,141.5575971},{0.1531755275,141.525463},{0.1534972838,141.4934024},{0.153819374,141.461583},{0.1541417982,141.4299457},{0.1544645563,141.3989194},{0.1547876484,141.3679407},{0.1551110744,141.3369396},{0.1554348343,141.3061831},{0.1557589282,141.2753304},{0.1560833561,141.2448593},{0.1564081179,141.2150258},{0.1567332136,141.184717},{0.1570586433,141.1551871},{0.1573844069,141.1255847},{0.1577105045,141.0961997},{0.158036936,141.0671744},{0.1583637015,141.0377446},{0.1586908009,141.0089625},{0.1590182342,140.9805454},{0.1593460015,140.952254},{0.1596741028,140.9236775},{0.160002538,140.8959711},{0.1603313071,140.8680822},{0.1606604102,140.840594},{0.1609898472,140.8133095},{0.1613196182,140.7862498},{0.1616497231,140.7599845},{0.161980162,140.7334515},{0.1623109348,140.7136048},{0.1626420415,140.6813128},{0.1629734822,140.6552853},{0.1633052569,140.6295219},{0.1636373655,140.604001},{0.163969808,140.5435822},{0.1643025845,140.5187445},{0.1646356949,140.5285929},{0.1649691393,140.5039895},{0.1653029176,140.4793339},{0.1656370299,140.4547695},{0.1659714761,140.4305886},{0.1663062562,140.4068171},{0.1666413703,140.3833144},{0.1669768184,140.3600062},{0.1673126003,140.33681},{0.1676487163,140.3469763},{0.1679851662,140.2912049},{0.16832195,140.2687794},{0.1686590677,140.2465824},{0.1689965195,140.2245728},{0.1693343051,140.2022927},{0.1696724247,140.1807708},{0.1700108783,140.1596037},{0.1703496658,140.1384292},{0.1706887872,140.117582},{0.1710282426,140.0970509},{0.1713680319,140.0765421},{0.1717081552,140.0564447},{0.1720486124,140.0542997},{0.1723894036,140.0159093},{0.1727305287,139.9963187},{0.1730719878,139.976825},{0.1734137808,139.9577365},{0.1737559077,139.9390176},{0.1740983686,139.9204618},{0.1744411634,139.9019767},{0.1747842922,139.883819},{0.1751277549,139.8662521},{0.1754715516,139.848622},{0.1758156822,139.8319856},{0.1761601468,139.8148654},{0.1765049453,139.7978539},{0.1768500778,139.7810795},{0.1771955442,139.7644917},{0.1775413445,139.7485294},{0.1778874788,139.732556},{0.178233947,139.7114066},{0.1785807492,139.7011905},{0.1789278853,139.6853526},{0.1792753554,139.6484434},{0.1796231594,139.6540771},{0.1799712974,139.6388912},{0.1803197693,139.6244177},{0.1806685752,139.6102887},{0.181017715,139.5962506},{0.1813671887,139.5824213},{0.1817169964,139.5690157},{0.182067138,139.5558397},{0.1824176136,139.5318097},{0.1827684231,139.5185379},{0.1831195666,139.5057438},{0.183471044,139.4937332},{0.1838228554,139.4934184},{0.1841750007,139.48177},{0.1845274799,139.4691891},{0.1848802931,139.4579579},{0.1852334403,139.4470521},{0.1855869214,139.4374377},{0.1859407364,139.4271105},{0.1862948854,139.4183489},{0.1866493683,139.4084727},{0.1870041852,139.3988302},{0.187359336,139.3892483},{0.1877148207,139.385209},{0.1880706394,139.3772996},{0.1884267921,139.3636938},{0.1887832787,139.3553764},{0.1891400992,139.3472442},{0.1894972537,139.3393551},{0.1898547421,139.3318098},{0.1902125645,139.3244291},{0.1905707208,139.3171834},{0.1909292111,139.3108126},{0.1912880353,139.3041807},{0.1916471935,139.2978028},{0.1920066856,139.2919002},{0.1923665116,139.2861998},{0.1927266716,139.2800471},{0.1930871656,139.2748138},{0.1934479934,139.2694992},{0.1938091553,139.2642054},{0.1941706511,139.2598178},{0.1945324808,139.255737},{0.1948946444,139.2513165},{0.195257142,139.2473906},{0.1956199736,139.2440114},{0.1959831391,139.2408897},{0.1963466385,139.2388394},{0.1967104719,139.2363344},{0.1970746393,139.221375},{0.1974391406,139.2193636},{0.1978039758,139.2174095},{0.198169145,139.2161214},{0.1985346481,139.2281204},{0.1989004851,139.2078981},{0.1992666561,139.2072955},{0.1996331611,139.2258174},{0.2,139.2261935}
$\endgroup$
  • 1
    $\begingroup$ I don't think I would call the errors in your data outliers as such (i.stack.imgur.com/h6MwF.png). There seems to be a small systematic offset that comes and goes. $\endgroup$ – Rahul Nov 17 '14 at 15:48
  • $\begingroup$ @RahulNarain Yes, I just show the worst data set here. Other data sets look really like sudden outliers. Anyway, considering the proportion of such points and the overall smooth trend, I need to somehow identify and eliminate them. $\endgroup$ – xiaohuamao Nov 17 '14 at 15:55
5
$\begingroup$

As you expect monotonic "smooth" behavior, a simple solution is to z-score the differences.

diff = data[[All, 2]] // Differences;
mn = Mean[diff]
std = StandardDeviation[diff]

(* 3 std is bad *)

bad = Position[diff, x_ /; Abs[x] > Abs[mn + 3 std]]

ListPlot[data, PlotRange -> All]
ListPlot[data[[bad // Flatten]], PlotStyle -> Red]
Show[%, %%]

This can also be changed to scan the data locally rather than calculate the statistics over the entire array.

$\endgroup$
  • $\begingroup$ Thanks. This is good. But please allow me to ask more. Your method is not innately suitable for some smooth monotonic convex curve, at lease to some extent. Due to nonlinearity, the diff might vary extensively. The more it varies, the less justifiable your method becomes, right? It becomes harder to pick out outliers in the less varying region since overwhelmed by largely varying region. For instance, your method fails to pick out the last but 5 point, which even violates monotonicity. $\endgroup$ – xiaohuamao Nov 19 '14 at 3:43
  • $\begingroup$ Hope to hear comment from you or anybody else. I myself guess we'd better divide the data into a few pieces and then apply this z-score method accordingly. To avoid outliers pinpointed at division boundaries, we might as well try different random divisions a few times. Anyway, it sounds no more than a workaround. $\endgroup$ – xiaohuamao Nov 19 '14 at 3:52
  • $\begingroup$ This method is ideal only for uniformly sampled quadratic curves where the derivative should be constant (i.e., the values of "diff" are all equal if the data is perfect). For smooth curves such as yours I often add the additional step of detrending the "diff" list prior to the mean and standard deviation calculation with a low order polynomial or appropriate functional form. Since the pattern test uses ABS, it does not test for monotonicity. $\endgroup$ – John Morganthau Nov 19 '14 at 23:35
3
$\begingroup$

A common approach to removing outliers is to use an order statistic filter. The simplest of these is the MedianFilter:

x = data[[All, 1]];
ySmoothed = MedianFilter[data[[All, 2]], 5];
ListPlot[Transpose[{x, ySmoothed}]]

enter image description here

$\endgroup$
2
$\begingroup$
 n = 20 (*even*)
 f = Interpolation[Transpose[{data[[All, 1]],
    Join[ data[[;; (n/2 - 1), 2]] , MovingAverage[data[[All, 2]], n] ,
          data[[-n/2 ;;, 2]] ]}]]

 GraphicsColumn[{ListPlot[data], 
     ListPlot[ Select[  data , Abs[f[#[[1]]] - #[[2]]] < .2 &  ] ]}]

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.