8
$\begingroup$

I have a dataset that looks as follows enter image description here Now, this is from an actual measurement (of the reflection coefficient of a microchip with four LC resonators connected in a periodic fashion) so there is obviously noise involved. For reference, this is what it should look like in theory (with different numerical parameters) enter image description here

So there's a few things about the data. First of all there is gaussian(?) noise around the 'mean'. This is not really an issue; I'm only really interested in the positions/sharpness/depths of the resonances, and the noise doesn't complicate it too much. I suppose that if it would be very problematic, one could use GaussianFilter? But then there's a more problematic issue, and that is also why I put accents around mean. The data is not in a straight line; there is some nonlinear trend over the range of the data. This is what my question is about; getting rid of this trend so that the data is on a straight line. Once that is done, getting it to have its mean at 1 is of course easy, and I'll have a dataset that can be fitted with the model.

My issue is that I don't really know where to start. I did some searching, but either my search terms are terrible or this hasn't really been discussed; it is quite specific after all. I'd like your advice on how to proceed, as personally I don't have any clue. Maybe I should fit the data with some general function? It does seem to be somewhat oscillatory, but not very nicely.

I also apologize for not providing some code to generate similar looking data. I would, if I had an idea of how to go about that, but I don't as I'm not sure what kind of function would generate this type of trend.

Instead I have uploaded the x-coordinates (Frequency in GHz) and the y-coordinates (complex reflection coefficient) here (Dropbox, should be safe and registration free to download it). I should definitely add that what is plotted above is the absolute value of the y-coordinates.

Note that it doesn't require registration. It'll pop up a registration dialog when you open the site, but you can close this, and then you can click the download button (although it seems greyed out, but it isn't). They write this here too dropbox.com/help/20

Improve this question
$\endgroup$
12
  • $\begingroup$ Does the theory give any idea on what the nonlinear trend might be? $\endgroup$
    – J. M.'s lack of A.I. ♦
    Jun 13, 2015 at 14:03
  • $\begingroup$ Ah, good point. This is one dataset of many, taken with a Vector Network Analyser. It's connected to the device with SMP cables, which are calibrated before doing the measurement. The calibration is supposed to take care of these trends, but in this case it didn't. So, does theory tell us something.. I'm not sure. I think the noise comes from the cables, and that it might have some oscillatory component to it, but beyond that I can't say. The reason I don't just remeasure is that it is done at liquid helium temperatures, so I'd rather post process. $\endgroup$
    – user129412
    Jun 13, 2015 at 14:12
  • $\begingroup$ Is the broad peak on the underlying signal symmetrical if plotted on a log scale on the x-axis, might it be a very wide resonance peak? Otherwise it looks like it might have a component which could be modelled as a piecewise linear function. Without data it's a little hard to say. $\endgroup$
    – image_doctor
    Jun 13, 2015 at 15:25
  • $\begingroup$ I believe you should post at least: one dataset, or go asking here instead $\endgroup$
    – Dr. belisarius
    Jun 13, 2015 at 15:30
  • $\begingroup$ Okay, let me think of a way to post the data. Is uploading it to some file sharing website okay? $\endgroup$
    – user129412
    Jun 13, 2015 at 15:36

2 Answers 2

Reset to default
10
$\begingroup$

Rather than modelling the non-linearity analytically, I've directly attempted to remove it. The longer trend features of the spectrum have been extracted by smoothing out most of the noise and signal, then subtracting these from the original spectrum.

A convenient way to do this is with an interpolation function, Interpolation, constructed from the smoothed spectrum. 

freqs = dat[[All, 1]];
vals =  Abs@dat[[All, 2]];
smoothedVals = MeanFilter[vals, 5000];
if = Interpolation[{freqs, smoothedVals}\[Transpose]];
trdat = {First@#, Last@# - if@First@#} & /@ ({freqs, vals}\[Transpose]);

I've chosen a value of 5000 to samples to smooth the spectrum with as this gives what looks like to me the 'nicest' non-linearity. (3000 samples produces a nicer spectrum, but the non-linearity looks less attractive, try the values for yourself ). Below is a plot of the interpolation function for a 5000 sample window:

ListPlot[{First@#, if@First@#} & /@ ({freqs, vals}\[Transpose]), AxesLabel -> {"Freq", "Amplitude"};

Mathematica graphics

Subtracting the non-linearity from the spectrum gives us this the spectrum with the non-linearity removed:

ListPlot[trdat, PlotRange -> All, AxesLabel -> {"Freq", "Amplitude"}]

Mathematica graphics

You can get a version of the spectrum with noise more centred on zero by reducing the number of samples to smooth by, but the non-linearity looks more disjointed and it disturbs the level of the response peaks.

If you drop the smoothing window size down to 100 samples you begin to see the level shift associated with too small a sample window and more noise, and the signal, appearing in the non-linearity:

Mathematica graphics

and

Mathematica graphics

Further low pass filtering of the spectrum shows what maybe some harmonic interaction:

ListPlot[{freqs, MeanFilter[trdat[[All, 2]], 50]}\[Transpose],

PlotRange -> All, AxesLabel -> {"Freq", "Amplitude"}]

Mathematica graphics

It may be worth noting that noise amplitude seems to increase with frequency, so it may not be Gaussian.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Hm, I think this might be a manual version of a highpass filter in the end, right? It certainly also does work, and it is a little more transparent (if you don't know digital filters all that well, like me) $\endgroup$
    – user129412
    Jun 13, 2015 at 19:45
  • $\begingroup$ Broadly yes, it's the subtraction of a low pass filtered signal which isolates the observed non-linearity. Maybe that will give you some insight into it's origin ? $\endgroup$
    – image_doctor
    Jun 13, 2015 at 19:51
  • $\begingroup$ Well, I'm very new to experimental physics (my first project really), so I'm not too familiar with all the complications yet. I'm sure it'll make a nice discussion with my supervisor. If I would guess, it is something with reflections in the cables. $\endgroup$
    – user129412
    Jun 13, 2015 at 20:34
  • $\begingroup$ I'm not sure which approach is better, in terms of the results. They both seem good to me. However, I didn't come up with the answer I posted below, someone else did, so I'll accept yours as it is a nice alternative. $\endgroup$
    – user129412
    Jun 13, 2015 at 20:39
  • $\begingroup$ I'm actually having trouble with the result from below. The peak height seems to be strongly changed, which is a problem. I'll have to re-examine it, see if it is fixable, and if not if your approach provides a better solution. $\endgroup$
    – user129412
    Jun 15, 2015 at 19:38
5
$\begingroup$

Credit goes to Ondřej Grover from the Signal Processing stackexchange. He suggested using a High pass filter, as it clearly seems to be some low frequency signal. His solution involved a Python script (found here) but something similar can be done with Mathematica.

Using a forward-backward method and some experimentation with cutoff values, we can use Mathematica's Highpassfilter to fix our data (which I originally defined as Sxx:

Sxxfiltered = 
  HighpassFilter[HighpassFilter[Abs[Sxx], 0.0003], 0.0003];

If we now look at what the data looks like, we see that instead of the curvey data we had before, we instead get a nice, straight line. enter image description here

As you can see on the y-axis the scale is no longer as before, but this can be corrected for without too much trouble.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.