2
$\begingroup$

I've a data set consisting of two columns, the first column is a date and the second is a numeric cumulative amount. I tried to create a time series, resample it and smooth but I couldn't manage to smooth it enough. I'm a noob with Mathematica. Can you help me to understand what I'm doing wrong?

enter image description here

data = Import[
    "https://1drv.ms/x/s!AiF0MVfYzFaAi3cO044kGlJW0Oam?e=Mt7qTv"][[1]];

data2 = data[[2 ;;, {1, 2}]] 

DateListPlot[data2]

ts = TimeSeries[data2, 
  ResamplingMethod -> {"Interpolation", InterpolationOrder -> 1}]

tr = TimeSeriesResample[ts, "Day", 
  ResamplingMethod -> {"Interpolation", InterpolationOrder -> 1}]

MovingMap[Median, tr, Quantity[3, "Months"]]

quotient[values_, times_] := 
 First[Differences[values]/Differences[times]]

mm = 
 MovingMap[quotient[#BoundaryValues, #BoundaryTimes] &, 
  tr, {.1, Right}]


DateListPlot[mm]
$\endgroup$
11
  • $\begingroup$ Look at your data set, the entries are not uniform and sometimes a comma is missing $\endgroup$ Commented Oct 23, 2020 at 9:30
  • 1
    $\begingroup$ Hi, life is hard, now some dates are missing. But I extracted only the cumulative values. You may smooth them e.g. by applying a filter. E.g. sd = GaussianFilter[dat, 20]; ListPlot[sd] . With the second argument to GaussianFilter you can adjust the strength of the filter. $\endgroup$ Commented Oct 23, 2020 at 14:02
  • 1
    $\begingroup$ Strange, when I looked at your Excel post, there were a lot of date fields like "xxxxx". I checked again but now it is o.k. No use of worrying about. $\endgroup$ Commented Oct 23, 2020 at 14:35
  • 1
    $\begingroup$ Yes you are right, GaussianFilter does not swallow TimesSeries but ordinary list of data. To apply a filter to TimeSeries look at MovingMap. $\endgroup$ Commented Oct 23, 2020 at 14:54
  • 2
    $\begingroup$ Not at all forbidden to cross-post. But it is good etiquette to mention that with cross-links in both places. The reason is that readers in one forum can become aware of what has already been stated/answered in the other. $\endgroup$ Commented Oct 23, 2020 at 15:19

1 Answer 1

5
$\begingroup$

You might want to consider kernel regression. (It would be great if Mathematica would offer a function to do so. And it's likely that someone has already produced a package for this.)

Kernel regression is similar to using a "moving mean" but with weighting where each data point's influence decreases with distance from that data point.

Below is a very crude implementation of that for your data and your particular question. Note that I've converted the dates to the number of days since January 1, 1900 as I just don't want to deal with date objects.

data = {{42005, -107}, {42019, -429}, {42025, -459.5}, {42031, -496.1}, {42033, -573.14}, {42034, -625.14}, {42035, -650.53}, {42036, -655.53}, {42037, -675.53}, {42038, -680.53}, {42039, -685.53}, {42040, -790.53}, {42051, -798.33}, {42057, -805.93}, {42058, -815.69}, {42061, -820.56}, {42064, -900.56}, {42066, -924.56}, {42067, -954.96}, {42076, -960.16}, {42081, -969.56}, {42085, -993.08}, {42089, -997.16}, {42092, -1002.16}, {42093, -1007.16}, {42094, -1014.66}, {42095, -1027.79}, {42097, -1038.05}, {42099, -1043.05}, {42102, -1053.05}, {42103, -1058.14}, {42104, -1068.32}, {42106, -1078.45}, {42107, -1091.82}, {42108, -1097.05}, {42109, -1155.68}, {42110, -1171.28}, {42111, -1185.91}, {42114, -1195.6}, {42115, -1231.6}, {42116, -1247.16}, {42117, -1252.16}, {42118, -1262.16}, {42121, -1390.86}, {42122, -1405.86}, {42123, -1457.86}, {42124, -1478.93}, {42128, -1487.93}, {42130, -1497.9}, {42131, -1503.42}, {42132, -1508.42}, {42135, -1513.42}, {42136, -1570.41}, {42144, -1575.61}, {42146, -1596.96}, {42148, -1603.96}, {42149, -1661.72}, {42151, -1666.72}, {42153, -1679.48}, {42157, -1692.24}, {42159, -1697.24}, {42160, -1715}, {42163, -1745}, {42164, -1750.52}, {42165, -1763.28}, {42167, -1776.04}, {42170, -1781.04}, {42171, -1793.8}, {42172, -1873.8}, {42173, -1906.8}, {42179, -1918.8}, {42180, -1923.8}, {42181, -1953.8}, {42184, -1972.08}, {42185, -2034.15}, {42186, -2076.91}, {42187, -2086.91}, {42188, -2191.91}, {42190, -2209.67}, {42191, -2223.06}, {42193, -2228.1}, {42195, -2272.88}, {42198, -2282.76}, {42200, -2295.52}, {42201, -2303.98}, {42207, -2309.07}, {42209, -2324.11}, {42212, -2241.91}, {42213, -2270.91}, {42214, -2305.91}, {42215, -2310.91}, {42227, -2334.91}, {42228, -2351.41}, {42229, -2361.41}, {42234, -2411.86}, {42237, -2423.86}, {42240, -2458.86}, {42241, -2463.86}, {42243, -2398.17}, {42244, -2406.17}, {42247, -2447.17}, {42250, -2490.17}, {42251, -2495.17}, {42252, -2515.17}, {42255, -2550.17}, {42256, -2581.17}, {42258, -2592.17}, {42260, -2595.62}, {42261, -2614.07}, {42262, -2624.07}, {42263, -2657.52}, {42264, -2662.52}, {42265, -2667.01}, {42269, -2701.5}, {42274, -2623.75}, {42275, -2642.31}, {42276, -2656.31}, {42277, -2689.87}, {42279, -2704.87}, {42280, -2712.87}, {42281, -2716.87}, {42282, -2721.87}, {42283, -2731.87}, {42286, -2736.87}, {42290, -2767.56}, {42291, -2797.56}, {42292, -2812.05}, {42298, -2827.05}, {42300, -2831.54}, {42303, -2846.54}, {42304, -2760.11}, {42305, -2779.6}, {42306, -2830.6}, {42310, -2834.6}, {42311, -2839.6}, {42312, -2852.6}, {42314, -2916.6}, {42315, -3061.49}, {42319, -3072.98}, {42320, -3089.48}, {42325, -3104.48}, {42327, -3116.56}, {42328, -3146.56}, {42331, -3149.86}, {42332, -3164.86}, {42334, -3168.86}, {42335, -3131.68}, {42338, -3146.68}, {42339, -3155.68}, {42341, -3082.68}, {42345, -2967.45}, {42347, -2969.45}, {42348, -2999.45}, {42352, -3026.05}, {42353, -2996.05}, {42360, -2999.05}, {42361, -3009.05}, {42364, -3014.05}, {42365, -2877.05}, {42372, -2909.05}, {42374, -2896.75}, {42375, -2906.75}, {42376, -2929.25}, {42378, -2934.25}, {42381, -2929.25}, {42383, -2953.5}, {42388, -2956.85}, {42389, -2966.85}, {42390, -2981.85}, {42391, -3045.85}, {42394, -3052.85}, {42395, -3057.85}, {42396, -2963.85}, {42397, -2982.38}, {42401, -2988.38}, {42403, -3022.38}, {42404, -3042.38}, {42406, -3113.71}, {42408, -3143.71}, {42409, -3058.71}, {42410, -3066.71}, {42411, -3096.71}, {42412, -3042.71}, {42416, -3018.71}, {42417, -3008.91}, {42421, -3016.91}, {42423, -2978.67}, {42425, -2986.67}, {42426, -2891.55}, {42427, -2726.55}, {42429, -2730.55}, {42430, -2740.55}, {42431, -2748.55}, {42432, -2807.55}, {42435, -2827.55}, {42436, -2836.55}, {42438, -2874.55}, {42441, -2889.55}, {42444, -2944.55}, {42445, -2954.55}, {42446, -2992.55}, {42447, -2999.05}, {42449, -3014.05}, {42452, -3154.05}, {42456, -2900.05}, {42459, -2989.35}, {42462, -2998.15}, {42464, -3023.15}, {42465, -3031.15}, {42467, -3048.15}, {42471, -3156.15}, {42472, -3180.15}, {42475, -3164.15}, {42478, -3185.15}, {42479, -3188.15}, {42480, -3212.67}, {42481, -3283.17}, {42484, -3305.17}, {42486, -3321.17}, {42487, -3013.79}, {42489, -3021.79}, {42491, -3044.19}, {42492, -3063.19}, {42493, -3099.19}, {42494, -3089.99}, {42495, -3122.99}, {42496, -3139.99}, {42500, -3146.99}, {42501, -3149.99}, {42503, -3239.99}, {42505, -3200.99}, {42506, -3203.99}, {42507, -3223.99}, {42508, -3323.99}, {42509, -3349.99}, {42511, -3357.99}, {42513, -3388.99}, {42514, -3333.99}, {42515, -3371.99}, {42516, -3098.99}, {42517, -3137.79}, {42522, -3148.79}, {42524, -3172.79}, {42526, -3180.79}, {42528, -3188.79}, {42529, -3192.29}, {42536, -3209.54}, {42537, -3239.54}, {42538, -3217.54}, {42540, -3219.54}, {42541, -3249.54}, {42548, -2995.54}, {42549, -3035.54}, {42551, -3040.54}, {42552, -3086.54}, {42556, -3096.54}, {42558, -3129.54}, {42560, -3179.54}, {42562, -3356.54}, {42563, -3361.54}, {42566, -3429.04}, {42567, -3434.04}, {42570, -3449.04}, {42576, -3466.54}, {42577, -3501.54}, {42578, -3299.54}, {42579, -3179.68}, {42583, -3223.68}, {42584, -3232.68}, {42585, -3272.68}, {42586, -3208}, {42591, -3211.56}, {42592, -3229.06}, {42593, -3245.14}, {42597, -3253.14}, {42600, -3292.14}, {42601, -3312.14}, {42602, -3257.14}, {42605, -3307.14}, {42606, -3257.14}, {42608, -3274.14}, {42609, -3106}, {42611, -3149}, {42612, -3164}, {42613, -3222}, {42615, -3232}, {42618, -3240}, {42619, -3276.5}, {42620, -3166.5}, {42621, -3315.5}, {42622, -3326}, {42623, -3391}, {42626, -3494}, {42629, -3501}, {42634, -3373}, {42635, -3384}, {42636, -3399}, {42639, -3314}, {42640, -3202}, {42644, -3219}, {42645, -3234}, {42647, -3268}, {42650, -3279}, {42651, -3304}, {42653, -3327}, {42655, -3182.62}, {42657, -3200.62}, {42660, -3218.12}, {42662, -3227.12}, {42663, -3243.12}, {42664, -3279.12}, {42665, -3296.12}, {42667, -3339.12}, {42668, -3357.12}, {42669, -3252.12}, {42670, -3051.58}, {42675, -3057.58}, {42676, -3066.58}, {42677, -3081.18}, {42678, -3086.18}, {42682, -2976.67}, {42685, -2984.67}, {42691, -3001.67}, {42695, -3013.75}, {42696, -3022.75}, {42697, -2967.75}, {42698, -2982.75}, {42701, -2755.75}, {42702, -2783.21}, {42705, -2816.21}, {42706, -2834.71}, {42707, -2842.71}, {42710, -2757.71}, {42711, -2765.71}, {42719, -2773.71}, {42720, -2785.71}, {42726, -2793.71}, {42730, -2743.71}, {42731, -2455.71}, {42735, -2520.43}, {42736, -2528.43}, {42739, -2538.43}, {42740, -2548.43}, {42743, -2563.43}, {42751, -2508.43}, {42752, -2458.43}, {42760, -2355.67}, {42762, -2158.67}, {42767, -2198.67}, {42768, -2143.67}, {42769, -2153.67}, {42774, -2098.67}, {42775, -2113.67}, {42776, -2123.67}, {42780, -2138.67}, {42782, -2045.67}, {42790, -2060.67}, {42792, -2068.67}, {42793, -1709.67}, {42794, -1718.67}, {42795, -1750.67}, {42797, -1776.67}, {42800, -1786.67}, {42802, -1793.67}, {42807, -1733.67}, {42814, -1669.67}, {42815, -1599.67}, {42817, -1606.67}, {42818, -1506.67}, {42819, -1538.91}, {42821, -1412.91}, {42822, -1441.41}, {42825, -1413.41}, {42831, -1423.41}, {42836, -1408.13}, {42852, -1093.13}, {42853, -1116.58}, {42857, -1115.58}, {42860, -795.58}, {42870, -805.58}, {42872, -750.58}, {42882, -483.58}, {42884, -514.45}, {42886, -492.45}, {42887, -478.45}, {42893, -388.45}, {42898, -393.45}, {42899, -343.45}, {42901, -293.45}, {42906, -143.59}, {42908, -72.74}, {42909, -102.74}, {42912, -126.2}, {42913, 396.8}, {42914, 384.8}, {42917, 363.8}, {42918, 354.8}, {42920, 334.8}, {42922, 325.8}, {42924, 319.8}, {42926, 359.8}, {42929, 351.8}, {42930, 363.8}, {42935, 393.8}, {42937, 385.8}, {42942, 362.34}, {42943, 556.34}, {42944, 550.34}, {42948, 545.34}, {42949, 437.68}, {42955, 429.18}, {42957, 421.18}, {42968, 410.18}, {42974, 445.18}, {42975, 420.72}, {42976, 416.72}, {42979, 384.41}, {42980, 378.41}, {42982, 286.41}, {42983, 341.41}, {42984, 309.41}, {42998, 304.41}, {43004, 284.41}, {43005, 329.41}, {43006, 323.95}, {43007, 320.95}, {43010, 281.95}, {43011, 275.95}, {43012, 239.45}, {43013, 223.45}, {43014, 219.95}, {43017, 214.95}, {43020, 614.95}, {43028, 576.95}, {43033, 553.95}, {43035, 802.95}, {43041, 686.67}, {43042, 685.67}, {43043, 660.67}, {43045, 632.67}, {43046, 596.67}, {43047, 566.67}, {43053, 558.67}, {43054, 511.67}, {43055, 686.67}, {43060, 821.67}, {43062, 1391.67}, {43063, 1431.67}, {43066, 1381.67}, {43069, 1351.67}, {43072, 1341.17}, {43073, 1311.17}, {43074, 1271.17}, {43075, 1246.67}, {43076, 1301.67}, {43080, 1292.37}, {43090, 1537.37}, {43091, 2046.87}, {43092, 2176.87}, {43101, 2150.87}, {43102, 2141.87}, {43103, 2114.87}, {43108, 2164.87}, {43109, 2158.87}, {43110, 2228.87}, {43115, 2297.87}, {43116, 2367.87}, {43118, 2337.87}, {43119, 2441.87}, {43122, 2874.87}, {43123, 3032.87}, {43125, 3307.87}, {43129, 3294.87}, {43131, 3283.27}, {43132, 3257.87}, {43136, 3160.77}, {43137, 3139.77}, {43138, 3239.77}, {43139, 3235.77}, {43143, 3218.77}, {43144, 3190.27}, {43146, 3140.27}, {43147, 3080.27}, {43151, 3030.27}, {43152, 3015.27}, {43153, 3691.27}, {43157, 3732.67}, {43159, 3637.67}, {43161, 3612.27}, {43164, 3608.27}, {43165, 3650.77}, {43166, 3625.77}, {43167, 3555.77}, {43168, 3550.07}, {43171, 3589.97}, {43173, 3562.47}, {43174, 3530.17}, {43177, 3506.97}, {43178, 3501.97}, {43179, 3445.47}, {43180, 3272.47}, {43181, 3999.17}, {43182, 3992.27}, {43185, 4033.27}, {43186, 3995.27}, {43187, 3989.27}, {43188, 3951.27}, {43189, 3937.27}, {43194, 3913.27}, {43195, 3885.27}, {43198, 3904.07}, {43200, 3914.07}, {43201, 3792.87}, {43203, 3764.67}, {43205, 3757.67}, {43208, 3751.87}, {43209, 3723.37}, {43212, 4823.37}, {43213, 4818.47}, {43217, 4809.47}, {43220, 4735.87}, {43222, 4703.87}, {43223, 4758.87}, {43224, 4809.87}, {43227, 4792.27}, {43228, 4779.87}, {43230, 4774.67}, {43234, 4721.97}, {43236, 4765.17}, {43239, 4802.17}, {43241, 4872.17}, {43242, 6095.17}, {43243, 6025.17}, {43245, 6129.87}, {43246, 6116.87}, {43248, 6096.87}, {43249, 6073.07}, {43250, 6148.77}, {43251, 6468.77}, {43252, 6462.87}, {43254, 6442.87}, {43255, 6346.87}, {43256, 6316.87}, {43257, 6306.87}, {43259, 6297.97}, {43262, 6413.97}, {43263, 6410.37}, {43265, 6531.17}, {43269, 6508.17}, {43270, 6491.09}, {43272, 6463.09}, {43273, 7645.09}, {43275, 7596.3}, {43277, 7585.4}, {43279, 7569.4}, {43282, 7561.4}, {43284, 7517.4}, {43285, 7512.4}, {43286, 7472.4}, {43294, 7459.9}, {43298, 7474.1}, {43299, 7458.87}, {43300, 7439.87}, {43304, 7434.47}, {43305, 8507.47}, {43306, 8488.68}, {43308, 8389.68}, {43310, 8389.68}, {43317, 8381.08}, {43318, 8353.08}, {43320, 8408.08}, {43328, 8403.28}, {43334, 9298.28}, {43336, 9259.49}, {43340, 8977.49}, {43341, 9079.49}, {43342, 9074.49}, {43346, 9070.69}, {43347, 9042.69}, {43351, 9033.29}, {43354, 8717.93}, {43363, 8784.93}, {43364, 8753.14}, {43367, 8969.44}, {43368, 8954.99}, {43369, 8954.99}, {43372, 8919.99}, {43374, 8863.79}, {43375, 8814.37}, {43392, 8820.87}, {43395, 8811.17}, {43396, 8999.56}, {43397, 8949.77}, {43399, 8949.77}, {43403, 8967.77}, {43406, 8932.77}, {43409, 8987.77}, {43412, 9099.77}, {43423, 9093.69}, {43425, 9274.64}, {43426, 9268.64}, {43428, 9219.85}, {43431, 9202.46}, {43432, 9172.46}, {43433, 9163.28}, {43434, 9158.41}, {43438, 9116.41}, {43444, 9089.41}, {43445, 9015.41}, {43451, 8945.41}, {43453, 8931.91}, {43458, 9220.12}, {43461, 9160.12}, {43467, 9113.37}, {43472, 9053.37}, {43480, 8985.37}, {43483, 9042.19}, {43484, 9118.01}, {43487, 9714.51}, {43489, 9692.66}, {43493, 9632.66}, {43494, 9623.91}, {43498, 9577.91}, {43504, 9499.91}, {43507, 9499.91}, {43515, 9421.46}, {43517, 10287.46}, {43518, 10376.46}, {43519, 10354.46}, {43520, 10309.27}, {43523, 10239.27}, {43524, 10229.32}, {43526, 10183.82}, {43530, 10100.47}, {43537, 10090.77}, {43538, 10084.47}, {43539, 9993.47}, {43546, 11100.47}, {43549, 11140.41}, {43551, 11126.41}, {43553, 11111.41}, {43556, 11102.41}, {43557, 10966.41}, {43558, 10962.81}, {43564, 11062.81}, {43566, 11036.81}, {43567, 11030.81}, {43568, 11024.81}, {43570, 10947.81}, {43573, 10929.51}, {43574, 10924.01}, {43578, 12545.01}, {43579, 12518.21}, {43580, 12510.41}, {43581, 12507.41}, {43583, 12504.41}, {43584, 12383.91}, {43585, 12377.91}, {43586, 12347.17}, {43588, 12242.17}, {43590, 12233.57}, {43600, 12179.57}, {43604, 12173.57}, {43606, 12073.57}, {43607, 13763.57}, {43610, 13718.38}, {43613, 13683.68}, {43614, 13674.41}, {43616, 13650.41}, {43619, 13566.28}, {43621, 13549.44}, {43628,  13541.44}, {43631, 13531.44}, {43632, 13520.22}, {43633, 13501.29}, {43635, 13492.02}, {43637, 13478.82}, {43638, 13470.82}, {43640, 15602.55}, {43642, 15586.35}, {43643, 15575.13}, {43644, 15537.13}, {43645, 15509.2}, {43648, 15476.76}, {43649, 15460.76}, {43651, 15386.76}, {43652, 15369.76}, {43654, 15313.36}, {43656, 15255.36}, {43657, 15241.56}, {43658, 15226.56}, {43661, 15216.06}, {43662, 15187.99}, {43664, 15175.19}, {43665, 15160.42}, {43666, 15110.8}, {43667, 15095.8}, {43668, 17467.8}, {43670, 17480.03}, {43672, 17474.53}, {43675, 17451.03}, {43677, 17371.03}, {43680, 17337.88}, {43683, 17322.88}, {43684, 17309.88}, {43685, 17641.88}, {43687, 17614.88}, {43696, 17599.84}, {43699, 19547.84}, {43701, 19493.71}, {43704, 19578.71}, {43705, 19562.01}, {43706, 19537.61}, {43707, 19162.61}, {43710, 19151.1}, {43711, 19062.7}, {43712, 19048.34}, {43713, 19033.11}, {43714, 19333.11}, {43716, 19313.14}, {43718, 19298.14}, {43720, 19289.24}, {43721, 19274.04}, {43726, 19263.56}, {43727, 19248.56}, {43729, 19234.2}, {43731, 19928.6}, {43732, 19866.37}, {43734, 19840.19}, {43737, 19767.5}, {43738, 19714.6}, {43739, 19673.8}, {43740, 19661.2}, {43741, 19581.2}, {43742, 19576.32}, {43743, 19567.32}, {43744, 19546.74}, {43745, 19528.74}, {43746, 19520.84}, {43747, 19470.84}, {43749, 19485.06}, {43752, 19428.06}, {43753, 19394.84}, {43754, 19382.84}, {43755, 19375.84}, {43757, 19360.86}, {43758, 19480.86}, {43759, 19649.66}, {43760, 19633.66}, {43761, 20604.66}, {43762, 20570.55}, {43763, 20554.55}, {43765, 20546.95}, {43766, 20155.95}, {43767, 20139.95}, {43768, 19894.95}, {43769, 19869.95}, {43771, 19855.95}, {43774, 19903.95}, {43777, 19891.95}, {43780, 19867.73}, {43785, 19685.73}, {43787, 19670.73}, {43792, 20305.73}, {43794, 20260.73}, {43795, 20363.73}, {43796, 20172.73}, {43801, 20118.58}, {43802, 20023.6}, {43808, 20018.4}, {43809, 19959.9}, {43812, 20159.9}, {43814, 20052.7}, {43815, 20027.2}, {43816, 20024.4}, {43817, 20013.4}, {43818, 19518.42}, {43821, 19514.07}, {43822, 20365.07}, {43823, 20315.94}, {43833, 20235.94}, {43834, 20231.94}, {43835, 20189.94}, {43838, 20114.94}, {43839, 20099.94}, {43843, 20092.14}, {43845, 19842.14}, {43849, 19820.16}, {43850, 19773.22}, {43851, 19711.53}, {43852, 19671.53}, {43853, 19672.93}, {43854, 19437}, {43857, 20219}, {43859, 19950.6}, {43863, 19915.92}, {43864, 19975.92}, {43866, 19948.03}, {43877, 19888.23}, {43878, 19778.23}, {43880, 19763.25}, {43881, 19761.75}, {43883, 20217.75}, {43885, 20156.62}, {43888, 20595.4}, {43891, 20330.4}, {43894, 20253.86}, {43900, 20253.86}, {43901, 20205.16}, {43904, 20196.46}, {43905, 20018.6}, {43906, 20002.8}, {43908, 19985.2}, {43909, 19608.02}, {43911, 19567.72}, {43913, 20614.42}, {43914, 20569.42}, {43915, 20545.42}, {43916, 20509.12}, {43917, 20484.62}, {43919, 20452.02}, {43920, 20447.52}, {43921, 20207.52}, {43923, 20202.67}, {43924, 20418.67}, {43925, 20396.67}, {43927, 20360.07}, {43929, 20352.07}, {43934, 20326.57}, {43935, 20304.94}, {43936, 20296.94}, {43937, 20418.94}, {43941, 20364.88}, {43942, 20324.88}, {43943, 20904.88}, {43946, 20879.88}, {43949, 20911.88}, {43950, 20876.68}, {43951, 20848.48}, {43953, 20772.48}, {43955, 20493.48}, {43956, 20478.5}, {43959, 20455.5}, {43960, 20436.5}, {43963, 20418.5}, {43964, 20403.5}, {43967, 20382.5}, {43972, 20362.5}, {43973, 20753.5}, {43976, 20618.5}, {43979, 20636.5}, {43980, 20580.5}, {43983, 20558.5}, {43987, 20482.5}, {43990, 20340.5}, {43998, 20002.5}, {44004, 20544.9}, {44010, 20770.9}, {44017, 20750.2}, {44020, 20735.2}, {44022, 20711.2}, {44027, 20461.2}, {44029, 20431.2}, {44031, 20388}, {44032, 20372}, {44034, 20563.93}, {44035, 20548.95}, {44040, 20999.35}, {44041, 20939.35}, {44045, 20906.35}, {44046, 20729.35}, {44048, 20724.85}, {44052, 20678.85}, {44055, 20637.85}, {44060, 20600.85}, {44062, 20558.85}, {44063, 20421.85}, {44064, 20361.85}, {44067, 20269.85}, {44070, 20181.85}, {44071, 20942.39}, {44072, 20899.99}, {44074, 20649.99}, {44077, 20573.99}, {44081, 20535.99}, {44086, 20605.99}, {44090, 20715.99}, {44092, 20660.99}};

I also want to note that your data has some odd features (like jumps and serial correlation). Consider the following segment of data:

ListPlot[{data, data}, PlotRange -> {{43500, 43600}, {8000, 13000}}, 
 Joined -> {True, False}, PlotStyle -> {Blue, {Blue, PointSize[0.02]}}]

Oddities in data

(* Define a weighting function and the kernel estimator for predictions at date x *)
weights[x_, bandwidth_] := Exp[-(data[[All, 1]] - x)^2/bandwidth^2]
f[x_, bandwidth_] := Total[data[[All, 2]] weights[x, bandwidth]]/Total[weights[x, bandwidth]]

(* Try bandwidth = 100 *)
smoothed = Table[{x, f[x, 100]}, {x, Min[data[[All, 1]]], Max[data[[All, 1]]]}];
ListPlot[smoothed]

Smoothed data using kernel regression

Now find the derivative:

g = D[Total[data[[All, 2]] weights[x, h]]/Total[weights[x, h]], x];
Plot[g, {x, Min[data[[All, 1]]], Max[data[[All, 1]]]}]

Derivative of kernel regression fit

There are "automatic" choices for the bandwidth (try the R package np) but your mileage may vary. In short, choosing an optimal bandwidth is maybe more of an art as it will also depend on the subject matter and how the data was collected as opposed to depending on just the data values alone.

$\endgroup$
2
  • 3
    $\begingroup$ One can get similar plots using DerivativeFilter or DifferentiatorFilter. As for adjusting smoothness, I am out of my depth there. $\endgroup$ Commented Oct 24, 2020 at 20:08
  • $\begingroup$ @DanielLichtblau Good to know about those functions! Thanks! $\endgroup$
    – JimB
    Commented Oct 25, 2020 at 6:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.