How to generate a good Fourier spectrum with incomplete data?

This problem is perhaps a little bit more mathematical than programmatic. I'm using Fourier to analyze my data. Suppose I have a data set

 datainv3 = InverseFourier[#^(-3) & /@ Range@1000]


Now I make up a data series with a -3 spectrum. Analyze it with Fourier:

ListLogLogPlot[{Abs[Fourier[datainv3]],
Abs[Fourier[datainv3[[1 ;; 999]]]],
Abs[Fourier[datainv3[[1 ;; 800]]]]}, Frame -> True]


This gives me:

The blue line looks fine. But what is wrong with the orange and green ones? Fourier doesn't return me a good spectrum, even if I use 99.9% data. Why will this happen?

Is there a way that I can still generate a good spectrum with incomplete data? Perhaps I should use something different rather than a simple Fourier? Because real data is never infinitely long, and I am afraid that Fourier will give me a very different result if I slightly vary the length of the signal fed into it——That's horrible.

I am sorry to mention this question again, because the problem is still NOT solved even after I make up the other 'half' spectrum as suggestted by @flinty.

considering the following code:

n = 1000;
spectrum = Range[n]^-3;
datainv3 = InverseFourier[Join[spectrum, Reverse@spectrum]];
Show[
LogLogPlot[x^-3, {x, 1, 2 n}, PlotStyle -> {Gray,Dashed},
Filling -> Bottom],
ListLogLogPlot[Abs[Fourier[datainv3[[;; 2 n - 1]]]], PlotStyle -> Red],
ListLogLogPlot[Re[Fourier[datainv3[[;; 2 n - 1]]]], PlotStyle -> Green]]


It gives this:

It is, emmm, I have to say, not much better than the original result.

It is a bit better if I Re datainv3 before taking the Fourier. The code reads:

n = 1000;
spectrum = Range[n]^-3;
datainv3 = InverseFourier[Join[spectrum, Reverse@spectrum]];
Show[
LogLogPlot[x^-3, {x, 1, 2 n}, PlotStyle -> {Gray, Dashed}, Filling -> Bottom],
ListLogLogPlot[Abs[Fourier[Re@datainv3[[;; 2 n - 1]]]], PlotStyle -> Red],
ListLogLogPlot[Re[Fourier[Re@datainv3[[;; 2 n - 1]]]], PlotStyle -> Green],
ListLogLogPlot[Re[Fourier[Re@datainv3[[;; 2 n - 10]]]], PlotStyle -> Pink]
]


It gives me this

It looks better than the above picture but can hardly be satisfying. datainv3 is 2000 numbers long. However, the Fourier spectrum varies significantly if I miss out on only 10 data points. Does this mean that the Fourier spectrums cannot be compared if they are from original data series with different lengths?

I mean, in my understanding, the spectrum of a signal is its intrinsic properties. The spectrum should not vary greatly if we analyze examples with different lengths. For example, the turbulent kinetic energy spectrum is a -5/3 power law (http://brennen.caltech.edu/fluidbook/basicfluiddynamics/turbulence/turbulencescales.pdf). It is a -5/3 power law if you observe turbulence for 10 minutes, and it is a -5/3 power law if you observe turbulence for 1 hour. Perhaps the low-frequency part may vary a bit as the sign length changes (longer signal length may introduce the lower frequency part), but the high-frequency part should remain the same because the latter represents the small-scale fluctuation.

• It doesn't seem wrong, but...it is very strange. Maybe someone with solid Fourier chops (read: not me) can explain and perhaps suggest a remedy. Commented Oct 12, 2023 at 17:15
• See my answer below. You need to drop a point before you reverse the data and join it to the original list.
– Hugh
Commented Oct 27, 2023 at 10:51
• These are wrong Re[Fourier[Re@datainv3[[;; 2 n - 10]]]] - it should be just Re[Fourier[datainv3]] Commented Oct 27, 2023 at 12:16
• @flinty I want to generate a spectrum using incomplete data. In my first image, Abs[Fourier[datainv3]] is a perfect blue line, even though I miss the other half of the spectrum when generating datainv3. However, Abs[Fourier[datainv3[[1;;-2]]]] looks awful. Then I ask, why didn't Fourier return me a good spectrum, even if I use 99.9% data. Commented Oct 27, 2023 at 13:42
• @flinty Emmmm.. I do not quite understand. ListLogLogPlot[Re[Fourier[Re@Join[datainv3[[;; n - 10]]]]]] doesn't give a good spectrum. Commented Oct 27, 2023 at 15:13

I think we are now getting to the true question the OP is asking. The question seems to be why is it if I chop a little bit of the end of my signal it makes such a big difference?

I am assuming that the signal is in the time domain and we have been given a representation in the frequency domain. The chopping is done in the time domain. We then compare the Fourier transform of the un-chopped and chopped signal in the frequency domain. OP please confirm that this the question you are asking?

Let's make the signal in the frequency domain and then get the time domain version; we have.

    a = #^(-3) & /@ Range@1000;
b = Join[a, Reverse[Rest[a]]];
c = InverseFourier[b];

ListLogLogPlot[a, PlotHighlighting -> None, PlotLabel -> "Frequency domain"]
ListLinePlot[c, PlotHighlighting -> None, PlotLabel -> "Time domain"]


Now chop the time domain, and go back to the frequency domain and compare

 c1 = c[[1 ;; -10]];
b1 = Fourier[c1];
ListLogLogPlot[Abs[{b, b1}], PlotHighlighting -> None,
Joined -> True,
PlotLegends -> LineLegend[{"Un-chopped", "Chopped"}]]


So just removing 10 points in the time domain has made a huge difference in the frequency domain.

A point to note is that the data in the time domain is large at the end of the data. This means that Fourier is calculating a Fourier series under the assumption that the data is periodic. When the signal is chopped there is a jump between the end of one period and the start of the next. We can see this if we make the time data periodic. Thus we make two periods from the time data and look at the join between the periods.

c2 = Join[c, c];
c3 = Join[c1, c1];
ListLinePlot[{c2[[2000 - 20 ;; 2000 + 20]],
c3[[2000 - 20 ;; 2000 + 20]]}, PlotHighlighting -> None,
PlotLegends -> LineLegend[{"Un-chopped", "Chopped"}]]


There is a clear jump in the data for the chopped case. This chop is modelled by high frequencies in the frequency domain which is clear from the frequency plot of the chopped data.

So the OP is asking how do we put the chopped data back. One approach could be to interpolate the missing data. I will have to think about this and come back

• Thank you very much. It is exactly what I ask. I guess that, in the real signal, such a "jump in the data" will not happen? Perhaps I should "detrend" my time-domain signal before taking the Fourier? Commented Oct 28, 2023 at 7:52
• Glad we got to the heart of your question. I agree that you need to look at the time history rather than the spectrum if you wish to compensate for truncating. If your time history is approaching zero at the end there will probably not be a problem. If there is a step then you may have an issue. Since I got to your issue you may wish to un-accepted the previous answer and accept mine!
– Hugh
Commented Oct 28, 2023 at 16:24

It's because your initial data that you pass into the inverse transform is missing the other 'half' before you invert it.

One annoying property of Fourier that continually trips people up is that it produces a symmetric spectrum when applied to real valued functions/data. Before you do the inverse, you are missing the other half. This is why the result of Fourier[InverseFourier[...]] contains unexpected imaginary stuff and presumably why you took the Abs. If you add the right hand part of the spectrum then it works, and the only imaginary stuff you'll see will be due to numerical error, hence why I'm taking the Re part.

n = 1000;
spectrum = Range[n]^-3;
datainv3 = InverseFourier[Join[spectrum, Reverse@Rest@spectrum]];
Show[LogLogPlot[x^-3, {x, 1, n}, PlotStyle -> {Gray, Dashed},
Filling -> Bottom],
ListLogLogPlot[Re[Fourier[datainv3][[;; n]]], PlotStyle -> Red]]


• ... and given all that, the reason why the curves appeared at the ends is because your Abs is biasing things upwards due to the imaginary part (which should not be there - up to numerical error - if you use a symmetric spectrum) Commented Oct 14, 2023 at 8:35
• I updated my question. It seems that Re[Fourier[datainv3[[;; 2n-10]]]] still looks strange. Commented Oct 27, 2023 at 9:56
• This is not quite right - see my answer
– Hugh
Commented Oct 27, 2023 at 10:19
• @Harry you need to take the part after the closing brace of the Fourier, not before. Also you shouldn't need to take this part at all so I don't know why you're doing it. The plot will then work except will have a kink in it after x=n because it's not defined there. pastebin.com/vrehjTGY Commented Oct 27, 2023 at 12:11
• @Hugh thanks, I changed it to Reverse@Rest@spectrum to drop the zero frequency from the other half. Commented Oct 27, 2023 at 12:14

Note that supplying the reverse is along the correct lines but not quite right. One point has to be dropped. Below I make the data call it a. Then I just join on the reverse which is b. I also drop one point and then join on the reversed data call it b1. Finally I plot the last 10 points of each list

a = #^(-3) & /@ Range@1000;
b = Join[a, Reverse[a]];
b1 = Join[a, Reverse[Rest[a]]]; ListLinePlot[{b[[-10 ;; -1]],
b1[[-10 ;; -1]]}, PlotHighlighting -> None, PlotRange -> All]



Note that b returns to the first point at the end and b1 does not, it has one less point. Now we do the inverse Fourier transform and plot both the real and imaginary parts of the results

inv = InverseFourier[N[b]];
inv1 = InverseFourier[N[b1]];
ListPlot[Re@{inv, inv1}, PlotHighlighting -> None]
ListPlot[Im@{inv, inv1}, PlotHighlighting -> None]


Note that without and with the dropped point have different real parts and that the imaginary part without the dropped point is significant. With the dropped point the imaginary part is negligible noise.

What is happening here? If we are going from the time domain to the frequency domain via Fourier then we have equally sampled points in the time domain and this results in a periodic function in the frequency domain. Also, the Fourier transform provides both positive and negative frequencies. Fourier calculates just one period. By tradition it does the positive frequencies first and then the negative frequencies -periodic so correct. Each positive frequency has a corresponding negative frequency except for the zero frequency. The zero frequency value does not have a corresponding negative value. Thus when defining data in the frequency domain the first point is the zero frequency - which must not be repeated. As the test function is very large for the zero frequency the effect of repeating it wrongly is significant and shows up.

Hope that helps.

• It is helpful but it seems still not quite right. Try ListLogLogPlot[{a,Re@Fourier@inv1[[1 ;; -10]]}, PlotRange -> {{1, 1000}, All}] you will find that it still looks strange. Commented Oct 27, 2023 at 11:11