# Filtering of an unwanted frequency through a Fourier Transform

I have a data set with evenly spaced data points. The plot is frequency vs. intensity. The overall shape of the plot is an upwards curve into a plateau, this cannot be seen in the data as this is an unimportant feature. There is also an oscillation in this curve. This can be seen in the plot.

There are a few things I would like to do here…

1. Find out the frequency of this oscillation via a Fourier transform. I have done this already but I have encountered some problems. There are large peaks at the very end of the Fourier transformed data, I believe this can be sorted by zero filling using PadLeft/Right. I have also had quite a few attempts at this but have so far been unsuccessful.

2. I would like to filter this oscillation (frequency) from the data and then re-plot the data with this frequency removed. Again, I have had a few attempts at this but to no avail. I also believe that the data can be back-converted using the inverse Fourier transform.

The data that I am using has been attached below:

{{2.96536, 0.104234}, {2.98246, 0.0969915}, {2.99966,
0.102057}, {3.01696, 0.0921243}, {3.03436, 0.119644}, {3.05186,
0.111209}, {3.06945, 0.114199}, {3.08716, 0.109548}, {3.10496,
0.131311}, {3.12286, 0.11789}, {3.14087, 0.136387}, {3.15898,
0.156646}, {3.1772, 0.14701}, {3.19552, 0.170584}, {3.21395,
0.135949}, {3.23248, 0.155617}, {3.25112, 0.169365}, {3.26987,
0.177859}, {3.28873, 0.166621}, {3.30769, 0.16418}, {3.32676,
0.176456}, {3.34595, 0.194153}, {3.36524, 0.191821}, {3.38465,
0.16664}, {3.40417, 0.19331}, {3.4238, 0.197461}, {3.44354,
0.190734}, {3.4634, 0.218241}, {3.48337, 0.22238}, {3.50346,
0.218784}, {3.52366, 0.215578}, {3.54398, 0.23948}, {3.56442,
0.245256}, {3.58497, 0.226847}, {3.60564, 0.219071}, {3.62643,
0.223373}, {3.64735, 0.225021}, {3.66838, 0.225226}, {3.68953,
0.233922}, {3.71081, 0.232157}, {3.73221, 0.228506}, {3.75373,
0.203525}, {3.77538, 0.244966}, {3.79715, 0.241911}, {3.81904,
0.21011}, {3.84107, 0.208428}, {3.86322, 0.227731}, {3.88549,
0.222106}, {3.9079, 0.237606}, {3.93043, 0.221255}, {3.9531,
0.19241}, {3.9759, 0.221645}, {3.99882, 0.243768}, {4.02188,
0.217034}, {4.04507, 0.203556}, {4.0684, 0.205594}, {4.09186,
0.224882}, {4.11546, 0.213087}, {4.13919, 0.205046}, {4.16306,
0.216099}, {4.18706, 0.225207}, {4.21121, 0.222689}, {4.23549,
0.214728}, {4.25992, 0.23614}, {4.28448, 0.240632}, {4.30919,
0.224024}, {4.33404, 0.239854}, {4.35903, 0.242658}, {4.38417,
0.27057}, {4.40945, 0.258658}, {4.43488, 0.265637}, {4.46045,
0.259903}, {4.48617, 0.269462}, {4.51204, 0.283}, {4.53806,
0.289011}, {4.56423, 0.30783}, {4.59055, 0.297366}, {4.61702,
0.299034}, {4.64365, 0.311518}, {4.67042, 0.305525}, {4.69736,
0.313848}, {4.72444, 0.330391}, {4.75169, 0.329848}, {4.77909,
0.322798}, {4.80665, 0.351296}, {4.83437, 0.347589}, {4.86224,
0.370153}, {4.89028, 0.35712}, {4.91848, 0.357503}, {4.94685,
0.369257}, {4.97537, 0.342726}, {5.00406, 0.361666}, {5.03292,
0.353577}, {5.06194, 0.361541}, {5.09113, 0.355642}, {5.12049,
0.356704}, {5.15002, 0.333525}, {5.17971, 0.373981}, {5.20958,
0.365135}, {5.23963, 0.354161}, {5.26984, 0.338223}, {5.30023,
0.332814}, {5.33079, 0.34522}, {5.36153, 0.339017}, {5.39245,
0.335138}, {5.42355, 0.304693}, {5.45482, 0.3446}, {5.48628,
0.302124}, {5.51791, 0.319763}, {5.54973, 0.322771}, {5.58174,
0.330913}, {5.61392, 0.32405}, {5.6463, 0.356941}, {5.67886,
0.334621}, {5.7116, 0.342564}, {5.74454, 0.371111}, {5.77767,
0.34261}, {5.81098, 0.388414}, {5.84449, 0.384681}, {5.8782,
0.399269}, {5.91209, 0.387332}, {5.94619, 0.384739}, {5.98048,
0.379754}, {6.01496, 0.400486}, {6.04965, 0.45052}, {6.08453,
0.421576}, {6.11962, 0.426608}, {6.15491, 0.426233}, {6.1904,
0.436149}, {6.2261, 0.455608}, {6.262, 0.45478}, {6.29811,
0.458522}, {6.33443, 0.471644}, {6.37096, 0.469424}, {6.4077,
0.450665}, {6.44465, 0.441694}, {6.48181, 0.4626}, {6.51919,
0.477626}, {6.55678, 0.435283}, {6.59459, 0.429262}, {6.63262,
0.44671}, {6.67087, 0.403679}, {6.70934, 0.444082}, {6.74803,
0.450073}, {6.78694, 0.442818}, {6.82608, 0.431519}, {6.86544,
0.429014}, {6.90503, 0.446844}, {6.94485, 0.439155}, {6.9849,
0.441625}, {7.02517, 0.448944}, {7.06569, 0.469756}, {7.10643,
0.467907}, {7.14741, 0.48434}, {7.18863, 0.505442}, {7.23008,
0.493814}, {7.27177, 0.480819}, {7.31371, 0.509904}, {7.35588,
0.510856}, {7.3983, 0.520773}, {7.44096, 0.539145}, {7.48387,
0.541484}, {7.52703, 0.552256}, {7.57043, 0.566155}, {7.61409,
0.579046}, {7.65799, 0.55074}, {7.70215, 0.560432}, {7.74657,
0.546139}, {7.79124, 0.568914}, {7.83617, 0.549678}, {7.88136,
0.520845}, {7.9268, 0.531402}, {7.97251, 0.529267}, {8.01849,
0.545642}, {8.06473, 0.540622}, {8.11123, 0.534639}, {8.15801,
0.527247}, {8.20505, 0.517973}, {8.25237, 0.5166}, {8.29995,
0.51883}, {8.34782, 0.534269}, {8.39595, 0.535702}, {8.44437,
0.537064}, {8.49307, 0.550537}, {8.54204, 0.552024}, {8.5913,
0.53197}, {8.64084, 0.576196}, {8.69067, 0.590832}, {8.74078,
0.602607}, {8.79119, 0.585699}, {8.84188, 0.578139}, {8.89287,
0.57396}, {8.94415, 0.588638}, {8.99573, 0.55418}, {9.0476,
0.578538}, {9.09978, 0.588798}, {9.15225, 0.618243}, {9.20503,
0.61227}, {9.25811, 0.623181}, {9.3115, 0.614365}, {9.36519,
0.582252}, {9.4192, 0.591002}, {9.47351, 0.582036}, {9.52814,
0.57551}, {9.58309, 0.579221}, {9.63835, 0.601598}, {9.69393,
0.583821}, {9.74983, 0.601753}, {9.80605, 0.616571}, {9.8626,
0.623343}, {9.91948, 0.625228}, {9.97668, 0.646208}, {10.0342,
0.640938}, {10.0921, 0.652611}, {10.1503, 0.662041}, {10.2088,
0.667227}, {10.2677, 0.676957}, {10.3269, 0.668459}, {10.3864,
0.676449}, {10.4463, 0.663062}, {10.5066, 0.671728}, {10.5671,
0.658277}, {10.6281, 0.645957}, {10.6894, 0.672471}, {10.751,
0.628736}, {10.813, 0.632566}, {10.8754, 0.636309}, {10.9381,
0.65611}, {11.0012, 0.615158}, {11.0646, 0.649256}, {11.1284,
0.632535}, {11.1926, 0.640524}, {11.2571, 0.649521}, {11.322,
0.678226}, {11.3873, 0.692686}, {11.453, 0.69979}, {11.519,
0.707412}, {11.5854, 0.702047}, {11.6523, 0.691649}, {11.7195,
0.70039}, {11.787, 0.708576}, {11.855, 0.688944}, {11.9234,
0.701633}, {11.9921, 0.679641}, {12.0613, 0.699239}, {12.1308,
0.673699}, {12.2008, 0.677758}, {12.2711, 0.671892}, {12.3419,
0.682907}, {12.4131, 0.689255}, {12.4847, 0.695831}, {12.5566,
0.708726}, {12.6291, 0.706843}, {12.7019, 0.700961}, {12.7751,
0.708156}, {12.8488, 0.735001}, {12.9229, 0.721247}, {12.9974,
0.727045}, {13.0724, 0.711983}, {13.1477, 0.747442}, {13.2236,
0.743774}, {13.2998, 0.703271}, {13.3765, 0.74117}, {13.4536,
0.739813}, {13.5312, 0.6996}, {13.6093, 0.698404}, {13.6877,
0.723868}, {13.7667, 0.709428}, {13.8461, 0.752924}, {13.9259,
0.747375}, {14.0062, 0.723421}, {14.087, 0.768379}, {14.1682,
0.756985}, {14.2499, 0.7739}, {14.3321, 0.780965}, {14.4147,
0.751474}, {14.4978, 0.797507}, {14.5814, 0.724096}, {14.6655,
0.744758}, {14.7501, 0.712119}, {14.8352, 0.740484}, {14.9207,
0.729569}, {15.0067, 0.696926}, {15.0933, 0.715435}, {15.1803,
0.716336}, {15.2679, 0.755648}, {15.3559, 0.785479}, {15.4445,
0.780429}, {15.5335, 0.810293}, {15.6231, 0.776618}, {15.7132,
0.806514}, {15.8038, 0.794361}, {15.8949, 0.768466}, {15.9866,
0.758793}, {16.0788, 0.799427}, {16.1715, 0.713711}, {16.2647,
0.779966}, {16.3585, 0.724277}, {16.4529, 0.743149}, {16.5477,
0.759417}, {16.6432, 0.783702}, {16.7391, 0.771348}, {16.8357,
0.833143}, {16.9328, 0.865346}, {17.0304, 0.838565}, {17.1286,
0.849294}, {17.2274, 0.772466}, {17.3267, 0.791529}, {17.4266,
0.768608}, {17.5271, 0.763986}, {17.6282, 0.741564}, {17.7299,
0.7283}, {17.8321, 0.748744}, {17.9349, 0.768283}, {18.0383,
0.803771}, {18.1424, 0.793137}, {18.247, 0.851768}, {18.3522,
0.887429}, {18.458, 0.864324}, {18.5645, 0.862401}, {18.6715,
0.789341}, {18.7792, 0.773965}, {18.8875, 0.827819}, {18.9964,
0.815678}, {19.106, 0.773111}, {19.2161, 0.849946}, {19.3269,
0.850296}, {19.4384, 0.834852}, {19.5505, 0.883059}, {19.6632,
0.881796}, {19.7766, 0.888633}, {19.8907, 0.862076}, {20.0054,
0.859558}, {20.1207, 0.852859}, {20.2367, 0.844693}, {20.3534,
0.792461}, {20.4708, 0.836719}, {20.5889, 0.834452}, {20.7076,
0.844025}, {20.827, 0.855403}, {20.9471, 0.864835}, {21.0679,
0.852573}, {21.1894, 0.88278}, {21.3116, 0.854075}, {21.4345,
0.823129}, {21.5581, 0.863766}, {21.6824, 0.787823}, {21.8074,
0.837579}, {21.9332, 0.832601}, {22.0596, 0.843718}, {22.1869,
0.8772}, {22.3148, 0.872614}, {22.4435, 0.884425}, {22.5729,
0.860168}, {22.7031, 0.843206}, {22.834, 0.861999}, {22.9657,
0.827936}, {23.0981, 0.842105}, {23.2313, 0.83025}, {23.3653,
0.843434}, {23.5, 0.866308}, {23.6355, 0.887162}, {23.7718,
0.879083}, {23.9089, 0.885303}, {24.0468, 0.872957}, {24.1854,
0.875103}, {24.3249, 0.851151}, {24.4652, 0.872132}, {24.6062,
0.893851}, {24.7481, 0.876361}, {24.8908, 0.891544}, {25.0344,
0.881993}, {25.1787, 0.867608}, {25.3239, 0.902699}, {25.47,
0.880534}, {25.6168, 0.883281}, {25.7646, 0.894037}, {25.9131,
0.865991}, {26.0626, 0.894638}, {26.2129, 0.884609}, {26.364,
0.902747}, {26.516, 0.894505}, {26.669, 0.891673}, {26.8227,
0.894257}, {26.9774, 0.868452}, {27.133, 0.894673}, {27.2894,
0.880052}, {27.4468, 0.89386}, {27.6051, 0.926104}, {27.7643,
0.89179}, {27.9244, 0.910141}, {28.0854, 0.889865}, {28.2474,
0.891172}, {28.4103, 0.873922}, {28.5741, 0.878855}, {28.7389,
0.890755}, {28.9046, 0.912344}, {29.0713, 0.904055}, {29.2389,
0.903534}, {29.4075, 0.892855}, {29.5771, 0.881583}}


I apologise for importing my data like this, if there is a more convenient way of doing so please let me know and I can import it that way.

Thank you very much for your help. Stuart.

For problem #2, I recommend a combination of TimeSeries and TimeSeriesResample to obtain a regular sampling of your data, followed by an eyeball inspection of the Periodogram. You'll see that there is a spike around 0.015-ish in this power spectrum.

You can then use a LowpassFilter or most filters from Linear and Nonlinear Filtering Reference

resampledData = TimeSeriesResample[data,
ResamplingMethod -> {"Interpolation", InterpolationOrder -> 3}];

Periodogram[resampledData[[All, 2]], PlotLabel -> "Periodogram Power Spectrum dB"]
lowpassed = LowpassFilter[TimeSeries[resampledData], 0.015]["Path"];
ListLinePlot[ {data, lowpassed }, PlotStyle -> {Gray, Red}]


EDIT: alternative method using the first 15 coefficients of a FourierDCT since we ran into unusual problems...

intp = Interpolation[data];
dt = Min[Differences[data][[2]]];
mint = Min[data[[All, 1]]];
maxt = Max[data[[All, 1]]];
resampled = Table[{t, intp[t]}, {t, mint, maxt, dt}];

ListLinePlot[{data,Transpose[{resampled[[All, 1]],
FourierDCT[
FourierDCT[resampled[[All, 2]]], 15],
Length[resampled]], "III"]
}]}, PlotStyle -> {Gray, Red}]

• Hi, thanks for your response. I am having a little problem with your answer. The red line does not seem to be fitting to the "old" data properly. The red line only has approximately half the intensity of the grey line. May 24, 2015 at 14:00
• @user14424 I don't understand, it looks a good fit to me. Would you rather the line rested on top of the peaks of the data? May 24, 2015 at 15:24
• Please see the addition at the bottom of the original question. As you can see the red line only has approximately half of the intensity as the grey line does. I am using exactly the same code that you produced. I'm not sure why the results are so drastically different. Thank you for your help again. May 24, 2015 at 15:34
• @user14424 can you post a screenshot of the notebook along with the graph? I tried starting from a clean notebook, setting data to your list and running my code, and I get the good result. May 24, 2015 at 15:42
• The screenshot has been added to the bottoms of the question as before. The top of the screenshot contains where I have opened a new notebook and set my data points equal to data. May 24, 2015 at 15:56

The data looks like it follows an exponential curve y=Ae^(x+t).

1. find a least-squares fit y=f(x) of the data on such a curve. So, find the function f(x) for the curve.
2. for each point on the graph do y0 = y-f(x). This flattens out the graph.
3. then do a FFT analysis on y0 to find the frequency.
4. filter out the frequency to find y0 -(FFT)-> y' and transform back using y'+f(x)

something like that?

• Hi, yes that's what I'd like to achieve. How would one do this in Mathematica? Thanks. May 24, 2015 at 17:34
• That's a pretty good candidate model. It makes the spike in the periodogram easier to see too...my attempt.. model = a - b Exp[-k *t]; fit = FindFit[data, model, {a, b, k}, t]; detrended = {#[[1]], #[[2]] - (model /. fit) /. t -> #[[1]]} & /@ data; GraphicsRow[{Show[ListPlot[data], Plot[model /. fit, {t, 0, 30}, PlotStyle -> Red], PlotLabel -> "Exp Model Fit"], ListLinePlot[detrended, PlotLabel -> "Model Detrended Data"], Periodogram[TimeSeriesResample[detrended][[All, 2]], PlotLabel -> "AC Periodogram"]}, ImageSize -> Large] May 24, 2015 at 18:18
• Hi, thank you for the help. I have now coded this up and extracted my desired frequency (which is correct). To filter out my frequency I simply changed the intensity of the corresponding data point to zero. I then used Re@InverseFourier at convert back again. Is this the best way of doing the filtering step? May 25, 2015 at 14:57