6
$\begingroup$

I have a data set with evenly spaced data points. The plot is frequency vs. intensity. The overall shape of the plot is an upwards curve into a plateau, this cannot be seen in the data as this is an unimportant feature. There is also an oscillation in this curve. This can be seen in the plot.

enter image description here

There are a few things I would like to do here…

  1. Find out the frequency of this oscillation via a Fourier transform. I have done this already but I have encountered some problems. There are large peaks at the very end of the Fourier transformed data, I believe this can be sorted by zero filling using PadLeft/Right. I have also had quite a few attempts at this but have so far been unsuccessful.

  2. I would like to filter this oscillation (frequency) from the data and then re-plot the data with this frequency removed. Again, I have had a few attempts at this but to no avail. I also believe that the data can be back-converted using the inverse Fourier transform.

The data that I am using has been attached below:

{{2.96536, 0.104234}, {2.98246, 0.0969915}, {2.99966, 
0.102057}, {3.01696, 0.0921243}, {3.03436, 0.119644}, {3.05186, 
0.111209}, {3.06945, 0.114199}, {3.08716, 0.109548}, {3.10496, 
0.131311}, {3.12286, 0.11789}, {3.14087, 0.136387}, {3.15898, 
0.156646}, {3.1772, 0.14701}, {3.19552, 0.170584}, {3.21395, 
0.135949}, {3.23248, 0.155617}, {3.25112, 0.169365}, {3.26987, 
0.177859}, {3.28873, 0.166621}, {3.30769, 0.16418}, {3.32676, 
0.176456}, {3.34595, 0.194153}, {3.36524, 0.191821}, {3.38465, 
0.16664}, {3.40417, 0.19331}, {3.4238, 0.197461}, {3.44354, 
0.190734}, {3.4634, 0.218241}, {3.48337, 0.22238}, {3.50346, 
0.218784}, {3.52366, 0.215578}, {3.54398, 0.23948}, {3.56442, 
0.245256}, {3.58497, 0.226847}, {3.60564, 0.219071}, {3.62643, 
0.223373}, {3.64735, 0.225021}, {3.66838, 0.225226}, {3.68953, 
0.233922}, {3.71081, 0.232157}, {3.73221, 0.228506}, {3.75373, 
0.203525}, {3.77538, 0.244966}, {3.79715, 0.241911}, {3.81904, 
0.21011}, {3.84107, 0.208428}, {3.86322, 0.227731}, {3.88549, 
0.222106}, {3.9079, 0.237606}, {3.93043, 0.221255}, {3.9531, 
0.19241}, {3.9759, 0.221645}, {3.99882, 0.243768}, {4.02188, 
0.217034}, {4.04507, 0.203556}, {4.0684, 0.205594}, {4.09186, 
0.224882}, {4.11546, 0.213087}, {4.13919, 0.205046}, {4.16306, 
0.216099}, {4.18706, 0.225207}, {4.21121, 0.222689}, {4.23549, 
0.214728}, {4.25992, 0.23614}, {4.28448, 0.240632}, {4.30919, 
0.224024}, {4.33404, 0.239854}, {4.35903, 0.242658}, {4.38417, 
0.27057}, {4.40945, 0.258658}, {4.43488, 0.265637}, {4.46045, 
0.259903}, {4.48617, 0.269462}, {4.51204, 0.283}, {4.53806, 
0.289011}, {4.56423, 0.30783}, {4.59055, 0.297366}, {4.61702, 
0.299034}, {4.64365, 0.311518}, {4.67042, 0.305525}, {4.69736, 
0.313848}, {4.72444, 0.330391}, {4.75169, 0.329848}, {4.77909, 
0.322798}, {4.80665, 0.351296}, {4.83437, 0.347589}, {4.86224, 
0.370153}, {4.89028, 0.35712}, {4.91848, 0.357503}, {4.94685, 
0.369257}, {4.97537, 0.342726}, {5.00406, 0.361666}, {5.03292, 
0.353577}, {5.06194, 0.361541}, {5.09113, 0.355642}, {5.12049, 
0.356704}, {5.15002, 0.333525}, {5.17971, 0.373981}, {5.20958, 
0.365135}, {5.23963, 0.354161}, {5.26984, 0.338223}, {5.30023, 
0.332814}, {5.33079, 0.34522}, {5.36153, 0.339017}, {5.39245, 
0.335138}, {5.42355, 0.304693}, {5.45482, 0.3446}, {5.48628, 
0.302124}, {5.51791, 0.319763}, {5.54973, 0.322771}, {5.58174, 
0.330913}, {5.61392, 0.32405}, {5.6463, 0.356941}, {5.67886, 
0.334621}, {5.7116, 0.342564}, {5.74454, 0.371111}, {5.77767, 
0.34261}, {5.81098, 0.388414}, {5.84449, 0.384681}, {5.8782, 
0.399269}, {5.91209, 0.387332}, {5.94619, 0.384739}, {5.98048, 
0.379754}, {6.01496, 0.400486}, {6.04965, 0.45052}, {6.08453, 
0.421576}, {6.11962, 0.426608}, {6.15491, 0.426233}, {6.1904, 
0.436149}, {6.2261, 0.455608}, {6.262, 0.45478}, {6.29811, 
0.458522}, {6.33443, 0.471644}, {6.37096, 0.469424}, {6.4077, 
0.450665}, {6.44465, 0.441694}, {6.48181, 0.4626}, {6.51919, 
0.477626}, {6.55678, 0.435283}, {6.59459, 0.429262}, {6.63262, 
0.44671}, {6.67087, 0.403679}, {6.70934, 0.444082}, {6.74803, 
0.450073}, {6.78694, 0.442818}, {6.82608, 0.431519}, {6.86544, 
0.429014}, {6.90503, 0.446844}, {6.94485, 0.439155}, {6.9849, 
0.441625}, {7.02517, 0.448944}, {7.06569, 0.469756}, {7.10643, 
0.467907}, {7.14741, 0.48434}, {7.18863, 0.505442}, {7.23008, 
0.493814}, {7.27177, 0.480819}, {7.31371, 0.509904}, {7.35588, 
0.510856}, {7.3983, 0.520773}, {7.44096, 0.539145}, {7.48387, 
0.541484}, {7.52703, 0.552256}, {7.57043, 0.566155}, {7.61409, 
0.579046}, {7.65799, 0.55074}, {7.70215, 0.560432}, {7.74657, 
0.546139}, {7.79124, 0.568914}, {7.83617, 0.549678}, {7.88136, 
0.520845}, {7.9268, 0.531402}, {7.97251, 0.529267}, {8.01849, 
0.545642}, {8.06473, 0.540622}, {8.11123, 0.534639}, {8.15801, 
0.527247}, {8.20505, 0.517973}, {8.25237, 0.5166}, {8.29995, 
0.51883}, {8.34782, 0.534269}, {8.39595, 0.535702}, {8.44437, 
0.537064}, {8.49307, 0.550537}, {8.54204, 0.552024}, {8.5913, 
0.53197}, {8.64084, 0.576196}, {8.69067, 0.590832}, {8.74078, 
0.602607}, {8.79119, 0.585699}, {8.84188, 0.578139}, {8.89287, 
0.57396}, {8.94415, 0.588638}, {8.99573, 0.55418}, {9.0476, 
0.578538}, {9.09978, 0.588798}, {9.15225, 0.618243}, {9.20503, 
0.61227}, {9.25811, 0.623181}, {9.3115, 0.614365}, {9.36519, 
0.582252}, {9.4192, 0.591002}, {9.47351, 0.582036}, {9.52814, 
0.57551}, {9.58309, 0.579221}, {9.63835, 0.601598}, {9.69393, 
0.583821}, {9.74983, 0.601753}, {9.80605, 0.616571}, {9.8626, 
0.623343}, {9.91948, 0.625228}, {9.97668, 0.646208}, {10.0342, 
0.640938}, {10.0921, 0.652611}, {10.1503, 0.662041}, {10.2088, 
0.667227}, {10.2677, 0.676957}, {10.3269, 0.668459}, {10.3864, 
0.676449}, {10.4463, 0.663062}, {10.5066, 0.671728}, {10.5671, 
0.658277}, {10.6281, 0.645957}, {10.6894, 0.672471}, {10.751, 
0.628736}, {10.813, 0.632566}, {10.8754, 0.636309}, {10.9381, 
0.65611}, {11.0012, 0.615158}, {11.0646, 0.649256}, {11.1284, 
0.632535}, {11.1926, 0.640524}, {11.2571, 0.649521}, {11.322, 
0.678226}, {11.3873, 0.692686}, {11.453, 0.69979}, {11.519, 
0.707412}, {11.5854, 0.702047}, {11.6523, 0.691649}, {11.7195, 
0.70039}, {11.787, 0.708576}, {11.855, 0.688944}, {11.9234, 
0.701633}, {11.9921, 0.679641}, {12.0613, 0.699239}, {12.1308, 
0.673699}, {12.2008, 0.677758}, {12.2711, 0.671892}, {12.3419, 
0.682907}, {12.4131, 0.689255}, {12.4847, 0.695831}, {12.5566, 
0.708726}, {12.6291, 0.706843}, {12.7019, 0.700961}, {12.7751, 
0.708156}, {12.8488, 0.735001}, {12.9229, 0.721247}, {12.9974, 
0.727045}, {13.0724, 0.711983}, {13.1477, 0.747442}, {13.2236, 
0.743774}, {13.2998, 0.703271}, {13.3765, 0.74117}, {13.4536, 
0.739813}, {13.5312, 0.6996}, {13.6093, 0.698404}, {13.6877, 
0.723868}, {13.7667, 0.709428}, {13.8461, 0.752924}, {13.9259, 
0.747375}, {14.0062, 0.723421}, {14.087, 0.768379}, {14.1682, 
0.756985}, {14.2499, 0.7739}, {14.3321, 0.780965}, {14.4147, 
0.751474}, {14.4978, 0.797507}, {14.5814, 0.724096}, {14.6655, 
0.744758}, {14.7501, 0.712119}, {14.8352, 0.740484}, {14.9207, 
0.729569}, {15.0067, 0.696926}, {15.0933, 0.715435}, {15.1803, 
0.716336}, {15.2679, 0.755648}, {15.3559, 0.785479}, {15.4445, 
0.780429}, {15.5335, 0.810293}, {15.6231, 0.776618}, {15.7132, 
0.806514}, {15.8038, 0.794361}, {15.8949, 0.768466}, {15.9866, 
0.758793}, {16.0788, 0.799427}, {16.1715, 0.713711}, {16.2647, 
0.779966}, {16.3585, 0.724277}, {16.4529, 0.743149}, {16.5477, 
0.759417}, {16.6432, 0.783702}, {16.7391, 0.771348}, {16.8357, 
0.833143}, {16.9328, 0.865346}, {17.0304, 0.838565}, {17.1286, 
0.849294}, {17.2274, 0.772466}, {17.3267, 0.791529}, {17.4266, 
0.768608}, {17.5271, 0.763986}, {17.6282, 0.741564}, {17.7299, 
0.7283}, {17.8321, 0.748744}, {17.9349, 0.768283}, {18.0383, 
0.803771}, {18.1424, 0.793137}, {18.247, 0.851768}, {18.3522, 
0.887429}, {18.458, 0.864324}, {18.5645, 0.862401}, {18.6715, 
0.789341}, {18.7792, 0.773965}, {18.8875, 0.827819}, {18.9964, 
0.815678}, {19.106, 0.773111}, {19.2161, 0.849946}, {19.3269, 
0.850296}, {19.4384, 0.834852}, {19.5505, 0.883059}, {19.6632, 
0.881796}, {19.7766, 0.888633}, {19.8907, 0.862076}, {20.0054, 
0.859558}, {20.1207, 0.852859}, {20.2367, 0.844693}, {20.3534, 
0.792461}, {20.4708, 0.836719}, {20.5889, 0.834452}, {20.7076, 
0.844025}, {20.827, 0.855403}, {20.9471, 0.864835}, {21.0679, 
0.852573}, {21.1894, 0.88278}, {21.3116, 0.854075}, {21.4345, 
0.823129}, {21.5581, 0.863766}, {21.6824, 0.787823}, {21.8074, 
0.837579}, {21.9332, 0.832601}, {22.0596, 0.843718}, {22.1869, 
0.8772}, {22.3148, 0.872614}, {22.4435, 0.884425}, {22.5729, 
0.860168}, {22.7031, 0.843206}, {22.834, 0.861999}, {22.9657, 
0.827936}, {23.0981, 0.842105}, {23.2313, 0.83025}, {23.3653, 
0.843434}, {23.5, 0.866308}, {23.6355, 0.887162}, {23.7718, 
0.879083}, {23.9089, 0.885303}, {24.0468, 0.872957}, {24.1854, 
0.875103}, {24.3249, 0.851151}, {24.4652, 0.872132}, {24.6062, 
0.893851}, {24.7481, 0.876361}, {24.8908, 0.891544}, {25.0344, 
0.881993}, {25.1787, 0.867608}, {25.3239, 0.902699}, {25.47, 
0.880534}, {25.6168, 0.883281}, {25.7646, 0.894037}, {25.9131, 
0.865991}, {26.0626, 0.894638}, {26.2129, 0.884609}, {26.364, 
0.902747}, {26.516, 0.894505}, {26.669, 0.891673}, {26.8227, 
0.894257}, {26.9774, 0.868452}, {27.133, 0.894673}, {27.2894, 
0.880052}, {27.4468, 0.89386}, {27.6051, 0.926104}, {27.7643, 
0.89179}, {27.9244, 0.910141}, {28.0854, 0.889865}, {28.2474, 
0.891172}, {28.4103, 0.873922}, {28.5741, 0.878855}, {28.7389, 
0.890755}, {28.9046, 0.912344}, {29.0713, 0.904055}, {29.2389, 
0.903534}, {29.4075, 0.892855}, {29.5771, 0.881583}}

I apologise for importing my data like this, if there is a more convenient way of doing so please let me know and I can import it that way.

Thank you very much for your help. Stuart.

Addition:

enter image description here

enter image description here

$\endgroup$
7
$\begingroup$

For problem #2, I recommend a combination of TimeSeries and TimeSeriesResample to obtain a regular sampling of your data, followed by an eyeball inspection of the Periodogram. You'll see that there is a spike around 0.015-ish in this power spectrum.

You can then use a LowpassFilter or most filters from Linear and Nonlinear Filtering Reference

resampledData = TimeSeriesResample[data,
 ResamplingMethod -> {"Interpolation", InterpolationOrder -> 3}];

Periodogram[resampledData[[All, 2]], PlotLabel -> "Periodogram Power Spectrum dB"]
lowpassed = LowpassFilter[TimeSeries[resampledData], 0.015]["Path"];
ListLinePlot[ {data, lowpassed }, PlotStyle -> {Gray, Red}]

enter image description here

EDIT: alternative method using the first 15 coefficients of a FourierDCT since we ran into unusual problems...

intp = Interpolation[data];
dt = Min[Differences[data][[2]]];
mint = Min[data[[All, 1]]];
maxt = Max[data[[All, 1]]];
resampled = Table[{t, intp[t]}, {t, mint, maxt, dt}];

ListLinePlot[{data,Transpose[{resampled[[All, 1]], 
 FourierDCT[
 PadRight[Take[
   FourierDCT[resampled[[All, 2]]], 15], 
  Length[resampled]], "III"]
}]}, PlotStyle -> {Gray, Red}]
$\endgroup$
  • $\begingroup$ Hi, thanks for your response. I am having a little problem with your answer. The red line does not seem to be fitting to the "old" data properly. The red line only has approximately half the intensity of the grey line. $\endgroup$ – user14424 May 24 '15 at 14:00
  • $\begingroup$ @user14424 I don't understand, it looks a good fit to me. Would you rather the line rested on top of the peaks of the data? $\endgroup$ – Histograms May 24 '15 at 15:24
  • $\begingroup$ Please see the addition at the bottom of the original question. As you can see the red line only has approximately half of the intensity as the grey line does. I am using exactly the same code that you produced. I'm not sure why the results are so drastically different. Thank you for your help again. $\endgroup$ – user14424 May 24 '15 at 15:34
  • $\begingroup$ @user14424 can you post a screenshot of the notebook along with the graph? I tried starting from a clean notebook, setting data to your list and running my code, and I get the good result. $\endgroup$ – Histograms May 24 '15 at 15:42
  • $\begingroup$ The screenshot has been added to the bottoms of the question as before. The top of the screenshot contains where I have opened a new notebook and set my data points equal to data. $\endgroup$ – user14424 May 24 '15 at 15:56
2
$\begingroup$

The data looks like it follows an exponential curve y=Ae^(x+t).

  1. find a least-squares fit y=f(x) of the data on such a curve. So, find the function f(x) for the curve.
  2. for each point on the graph do y0 = y-f(x). This flattens out the graph.
  3. then do a FFT analysis on y0 to find the frequency.
  4. filter out the frequency to find y0 -(FFT)-> y' and transform back using y'+f(x)

something like that?

$\endgroup$
  • $\begingroup$ Hi, yes that's what I'd like to achieve. How would one do this in Mathematica? Thanks. $\endgroup$ – user14424 May 24 '15 at 17:34
  • 1
    $\begingroup$ That's a pretty good candidate model. It makes the spike in the periodogram easier to see too...my attempt.. model = a - b Exp[-k *t]; fit = FindFit[data, model, {a, b, k}, t]; detrended = {#[[1]], #[[2]] - (model /. fit) /. t -> #[[1]]} & /@ data; GraphicsRow[{Show[ListPlot[data], Plot[model /. fit, {t, 0, 30}, PlotStyle -> Red], PlotLabel -> "Exp Model Fit"], ListLinePlot[detrended, PlotLabel -> "Model Detrended Data"], Periodogram[TimeSeriesResample[detrended][[All, 2]], PlotLabel -> "AC Periodogram"]}, ImageSize -> Large] $\endgroup$ – Histograms May 24 '15 at 18:18
  • 1
    $\begingroup$ Hi, thank you for the help. I have now coded this up and extracted my desired frequency (which is correct). To filter out my frequency I simply changed the intensity of the corresponding data point to zero. I then used Re@InverseFourier at convert back again. Is this the best way of doing the filtering step? $\endgroup$ – user14424 May 25 '15 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.