Filtering of an unwanted frequency through a Fourier Transform

Question

I have a data set with evenly spaced data points. The plot is frequency vs. intensity. The overall shape of the plot is an upwards curve into a plateau, this cannot be seen in the data as this is an unimportant feature. There is also an oscillation in this curve. This can be seen in the plot.

There are a few things I would like to do here…

1. Find out the frequency of this oscillation via a Fourier transform. I have done this already but I have encountered some problems. There are large peaks at the very end of the Fourier transformed data, I believe this can be sorted by zero filling using PadLeft/Right. I have also had quite a few attempts at this but have so far been unsuccessful.

2. I would like to filter this oscillation (frequency) from the data and then re-plot the data with this frequency removed. Again, I have had a few attempts at this but to no avail. I also believe that the data can be back-converted using the inverse Fourier transform.

The data that I am using has been attached below:

{{2.96536, 0.104234}, {2.98246, 0.0969915}, {2.99966, 
0.102057}, {3.01696, 0.0921243}, {3.03436, 0.119644}, {3.05186, 
0.111209}, {3.06945, 0.114199}, {3.08716, 0.109548}, {3.10496, 
0.131311}, {3.12286, 0.11789}, {3.14087, 0.136387}, {3.15898, 
0.156646}, {3.1772, 0.14701}, {3.19552, 0.170584}, {3.21395, 
0.135949}, {3.23248, 0.155617}, {3.25112, 0.169365}, {3.26987, 
0.177859}, {3.28873, 0.166621}, {3.30769, 0.16418}, {3.32676, 
0.176456}, {3.34595, 0.194153}, {3.36524, 0.191821}, {3.38465, 
0.16664}, {3.40417, 0.19331}, {3.4238, 0.197461}, {3.44354, 
0.190734}, {3.4634, 0.218241}, {3.48337, 0.22238}, {3.50346, 
0.218784}, {3.52366, 0.215578}, {3.54398, 0.23948}, {3.56442, 
0.245256}, {3.58497, 0.226847}, {3.60564, 0.219071}, {3.62643, 
0.223373}, {3.64735, 0.225021}, {3.66838, 0.225226}, {3.68953, 
0.233922}, {3.71081, 0.232157}, {3.73221, 0.228506}, {3.75373, 
0.203525}, {3.77538, 0.244966}, {3.79715, 0.241911}, {3.81904, 
0.21011}, {3.84107, 0.208428}, {3.86322, 0.227731}, {3.88549, 
0.222106}, {3.9079, 0.237606}, {3.93043, 0.221255}, {3.9531, 
0.19241}, {3.9759, 0.221645}, {3.99882, 0.243768}, {4.02188, 
0.217034}, {4.04507, 0.203556}, {4.0684, 0.205594}, {4.09186, 
0.224882}, {4.11546, 0.213087}, {4.13919, 0.205046}, {4.16306, 
0.216099}, {4.18706, 0.225207}, {4.21121, 0.222689}, {4.23549, 
0.214728}, {4.25992, 0.23614}, {4.28448, 0.240632}, {4.30919, 
0.224024}, {4.33404, 0.239854}, {4.35903, 0.242658}, {4.38417, 
0.27057}, {4.40945, 0.258658}, {4.43488, 0.265637}, {4.46045, 
0.259903}, {4.48617, 0.269462}, {4.51204, 0.283}, {4.53806, 
0.289011}, {4.56423, 0.30783}, {4.59055, 0.297366}, {4.61702, 
0.299034}, {4.64365, 0.311518}, {4.67042, 0.305525}, {4.69736, 
0.313848}, {4.72444, 0.330391}, {4.75169, 0.329848}, {4.77909, 
0.322798}, {4.80665, 0.351296}, {4.83437, 0.347589}, {4.86224, 
0.370153}, {4.89028, 0.35712}, {4.91848, 0.357503}, {4.94685, 
0.369257}, {4.97537, 0.342726}, {5.00406, 0.361666}, {5.03292, 
0.353577}, {5.06194, 0.361541}, {5.09113, 0.355642}, {5.12049, 
0.356704}, {5.15002, 0.333525}, {5.17971, 0.373981}, {5.20958, 
0.365135}, {5.23963, 0.354161}, {5.26984, 0.338223}, {5.30023, 
0.332814}, {5.33079, 0.34522}, {5.36153, 0.339017}, {5.39245, 
0.335138}, {5.42355, 0.304693}, {5.45482, 0.3446}, {5.48628, 
0.302124}, {5.51791, 0.319763}, {5.54973, 0.322771}, {5.58174, 
0.330913}, {5.61392, 0.32405}, {5.6463, 0.356941}, {5.67886, 
0.334621}, {5.7116, 0.342564}, {5.74454, 0.371111}, {5.77767, 
0.34261}, {5.81098, 0.388414}, {5.84449, 0.384681}, {5.8782, 
0.399269}, {5.91209, 0.387332}, {5.94619, 0.384739}, {5.98048, 
0.379754}, {6.01496, 0.400486}, {6.04965, 0.45052}, {6.08453, 
0.421576}, {6.11962, 0.426608}, {6.15491, 0.426233}, {6.1904, 
0.436149}, {6.2261, 0.455608}, {6.262, 0.45478}, {6.29811, 
0.458522}, {6.33443, 0.471644}, {6.37096, 0.469424}, {6.4077, 
0.450665}, {6.44465, 0.441694}, {6.48181, 0.4626}, {6.51919, 
0.477626}, {6.55678, 0.435283}, {6.59459, 0.429262}, {6.63262, 
0.44671}, {6.67087, 0.403679}, {6.70934, 0.444082}, {6.74803, 
0.450073}, {6.78694, 0.442818}, {6.82608, 0.431519}, {6.86544, 
0.429014}, {6.90503, 0.446844}, {6.94485, 0.439155}, {6.9849, 
0.441625}, {7.02517, 0.448944}, {7.06569, 0.469756}, {7.10643, 
0.467907}, {7.14741, 0.48434}, {7.18863, 0.505442}, {7.23008, 
0.493814}, {7.27177, 0.480819}, {7.31371, 0.509904}, {7.35588, 
0.510856}, {7.3983, 0.520773}, {7.44096, 0.539145}, {7.48387, 
0.541484}, {7.52703, 0.552256}, {7.57043, 0.566155}, {7.61409, 
0.579046}, {7.65799, 0.55074}, {7.70215, 0.560432}, {7.74657, 
0.546139}, {7.79124, 0.568914}, {7.83617, 0.549678}, {7.88136, 
0.520845}, {7.9268, 0.531402}, {7.97251, 0.529267}, {8.01849, 
0.545642}, {8.06473, 0.540622}, {8.11123, 0.534639}, {8.15801, 
0.527247}, {8.20505, 0.517973}, {8.25237, 0.5166}, {8.29995, 
0.51883}, {8.34782, 0.534269}, {8.39595, 0.535702}, {8.44437, 
0.537064}, {8.49307, 0.550537}, {8.54204, 0.552024}, {8.5913, 
0.53197}, {8.64084, 0.576196}, {8.69067, 0.590832}, {8.74078, 
0.602607}, {8.79119, 0.585699}, {8.84188, 0.578139}, {8.89287, 
0.57396}, {8.94415, 0.588638}, {8.99573, 0.55418}, {9.0476, 
0.578538}, {9.09978, 0.588798}, {9.15225, 0.618243}, {9.20503, 
0.61227}, {9.25811, 0.623181}, {9.3115, 0.614365}, {9.36519, 
0.582252}, {9.4192, 0.591002}, {9.47351, 0.582036}, {9.52814, 
0.57551}, {9.58309, 0.579221}, {9.63835, 0.601598}, {9.69393, 
0.583821}, {9.74983, 0.601753}, {9.80605, 0.616571}, {9.8626, 
0.623343}, {9.91948, 0.625228}, {9.97668, 0.646208}, {10.0342, 
0.640938}, {10.0921, 0.652611}, {10.1503, 0.662041}, {10.2088, 
0.667227}, {10.2677, 0.676957}, {10.3269, 0.668459}, {10.3864, 
0.676449}, {10.4463, 0.663062}, {10.5066, 0.671728}, {10.5671, 
0.658277}, {10.6281, 0.645957}, {10.6894, 0.672471}, {10.751, 
0.628736}, {10.813, 0.632566}, {10.8754, 0.636309}, {10.9381, 
0.65611}, {11.0012, 0.615158}, {11.0646, 0.649256}, {11.1284, 
0.632535}, {11.1926, 0.640524}, {11.2571, 0.649521}, {11.322, 
0.678226}, {11.3873, 0.692686}, {11.453, 0.69979}, {11.519, 
0.707412}, {11.5854, 0.702047}, {11.6523, 0.691649}, {11.7195, 
0.70039}, {11.787, 0.708576}, {11.855, 0.688944}, {11.9234, 
0.701633}, {11.9921, 0.679641}, {12.0613, 0.699239}, {12.1308, 
0.673699}, {12.2008, 0.677758}, {12.2711, 0.671892}, {12.3419, 
0.682907}, {12.4131, 0.689255}, {12.4847, 0.695831}, {12.5566, 
0.708726}, {12.6291, 0.706843}, {12.7019, 0.700961}, {12.7751, 
0.708156}, {12.8488, 0.735001}, {12.9229, 0.721247}, {12.9974, 
0.727045}, {13.0724, 0.711983}, {13.1477, 0.747442}, {13.2236, 
0.743774}, {13.2998, 0.703271}, {13.3765, 0.74117}, {13.4536, 
0.739813}, {13.5312, 0.6996}, {13.6093, 0.698404}, {13.6877, 
0.723868}, {13.7667, 0.709428}, {13.8461, 0.752924}, {13.9259, 
0.747375}, {14.0062, 0.723421}, {14.087, 0.768379}, {14.1682, 
0.756985}, {14.2499, 0.7739}, {14.3321, 0.780965}, {14.4147, 
0.751474}, {14.4978, 0.797507}, {14.5814, 0.724096}, {14.6655, 
0.744758}, {14.7501, 0.712119}, {14.8352, 0.740484}, {14.9207, 
0.729569}, {15.0067, 0.696926}, {15.0933, 0.715435}, {15.1803, 
0.716336}, {15.2679, 0.755648}, {15.3559, 0.785479}, {15.4445, 
0.780429}, {15.5335, 0.810293}, {15.6231, 0.776618}, {15.7132, 
0.806514}, {15.8038, 0.794361}, {15.8949, 0.768466}, {15.9866, 
0.758793}, {16.0788, 0.799427}, {16.1715, 0.713711}, {16.2647, 
0.779966}, {16.3585, 0.724277}, {16.4529, 0.743149}, {16.5477, 
0.759417}, {16.6432, 0.783702}, {16.7391, 0.771348}, {16.8357, 
0.833143}, {16.9328, 0.865346}, {17.0304, 0.838565}, {17.1286, 
0.849294}, {17.2274, 0.772466}, {17.3267, 0.791529}, {17.4266, 
0.768608}, {17.5271, 0.763986}, {17.6282, 0.741564}, {17.7299, 
0.7283}, {17.8321, 0.748744}, {17.9349, 0.768283}, {18.0383, 
0.803771}, {18.1424, 0.793137}, {18.247, 0.851768}, {18.3522, 
0.887429}, {18.458, 0.864324}, {18.5645, 0.862401}, {18.6715, 
0.789341}, {18.7792, 0.773965}, {18.8875, 0.827819}, {18.9964, 
0.815678}, {19.106, 0.773111}, {19.2161, 0.849946}, {19.3269, 
0.850296}, {19.4384, 0.834852}, {19.5505, 0.883059}, {19.6632, 
0.881796}, {19.7766, 0.888633}, {19.8907, 0.862076}, {20.0054, 
0.859558}, {20.1207, 0.852859}, {20.2367, 0.844693}, {20.3534, 
0.792461}, {20.4708, 0.836719}, {20.5889, 0.834452}, {20.7076, 
0.844025}, {20.827, 0.855403}, {20.9471, 0.864835}, {21.0679, 
0.852573}, {21.1894, 0.88278}, {21.3116, 0.854075}, {21.4345, 
0.823129}, {21.5581, 0.863766}, {21.6824, 0.787823}, {21.8074, 
0.837579}, {21.9332, 0.832601}, {22.0596, 0.843718}, {22.1869, 
0.8772}, {22.3148, 0.872614}, {22.4435, 0.884425}, {22.5729, 
0.860168}, {22.7031, 0.843206}, {22.834, 0.861999}, {22.9657, 
0.827936}, {23.0981, 0.842105}, {23.2313, 0.83025}, {23.3653, 
0.843434}, {23.5, 0.866308}, {23.6355, 0.887162}, {23.7718, 
0.879083}, {23.9089, 0.885303}, {24.0468, 0.872957}, {24.1854, 
0.875103}, {24.3249, 0.851151}, {24.4652, 0.872132}, {24.6062, 
0.893851}, {24.7481, 0.876361}, {24.8908, 0.891544}, {25.0344, 
0.881993}, {25.1787, 0.867608}, {25.3239, 0.902699}, {25.47, 
0.880534}, {25.6168, 0.883281}, {25.7646, 0.894037}, {25.9131, 
0.865991}, {26.0626, 0.894638}, {26.2129, 0.884609}, {26.364, 
0.902747}, {26.516, 0.894505}, {26.669, 0.891673}, {26.8227, 
0.894257}, {26.9774, 0.868452}, {27.133, 0.894673}, {27.2894, 
0.880052}, {27.4468, 0.89386}, {27.6051, 0.926104}, {27.7643, 
0.89179}, {27.9244, 0.910141}, {28.0854, 0.889865}, {28.2474, 
0.891172}, {28.4103, 0.873922}, {28.5741, 0.878855}, {28.7389, 
0.890755}, {28.9046, 0.912344}, {29.0713, 0.904055}, {29.2389, 
0.903534}, {29.4075, 0.892855}, {29.5771, 0.881583}}

I apologise for importing my data like this, if there is a more convenient way of doing so please let me know and I can import it that way.

Thank you very much for your help. Stuart.

Addition:

Answer 1

For problem #2, I recommend a combination of TimeSeries and TimeSeriesResample to obtain a regular sampling of your data, followed by an eyeball inspection of the Periodogram. You'll see that there is a spike around 0.015-ish in this power spectrum.

You can then use a LowpassFilter or most filters from Linear and Nonlinear Filtering Reference

resampledData = TimeSeriesResample[data,
 ResamplingMethod -> {"Interpolation", InterpolationOrder -> 3}];

Periodogram[resampledData[[All, 2]], PlotLabel -> "Periodogram Power Spectrum dB"]
lowpassed = LowpassFilter[TimeSeries[resampledData], 0.015]["Path"];
ListLinePlot[ {data, lowpassed }, PlotStyle -> {Gray, Red}]

EDIT: alternative method using the first 15 coefficients of a FourierDCT since we ran into unusual problems...

intp = Interpolation[data];
dt = Min[Differences[data][[2]]];
mint = Min[data[[All, 1]]];
maxt = Max[data[[All, 1]]];
resampled = Table[{t, intp[t]}, {t, mint, maxt, dt}];

ListLinePlot[{data,Transpose[{resampled[[All, 1]], 
 FourierDCT[
 PadRight[Take[
   FourierDCT[resampled[[All, 2]]], 15], 
  Length[resampled]], "III"]
}]}, PlotStyle -> {Gray, Red}]

Answer 2

The data looks like it follows an exponential curve y=Ae^(x+t).

1. find a least-squares fit y=f(x) of the data on such a curve. So, find the function f(x) for the curve.
2. for each point on the graph do y0 = y-f(x). This flattens out the graph.
3. then do a FFT analysis on y0 to find the frequency.
4. filter out the frequency to find y0 -(FFT)-> y' and transform back using y'+f(x)

something like that?