How to generate a good Fourier spectrum with incomplete data?

Question

This problem is perhaps a little bit more mathematical than programmatic. I'm using Fourier to analyze my data. Suppose I have a data set

 datainv3 = InverseFourier[#^(-3) & /@ Range@1000]

Now I make up a data series with a -3 spectrum. Analyze it with Fourier:

ListLogLogPlot[{Abs[Fourier[datainv3]], 
Abs[Fourier[datainv3[[1 ;; 999]]]], 
Abs[Fourier[datainv3[[1 ;; 800]]]]}, Frame -> True]

This gives me:

The blue line looks fine. But what is wrong with the orange and green ones? Fourier doesn't return me a good spectrum, even if I use 99.9% data. Why will this happen?

Is there a way that I can still generate a good spectrum with incomplete data? Perhaps I should use something different rather than a simple Fourier? Because real data is never infinitely long, and I am afraid that Fourier will give me a very different result if I slightly vary the length of the signal fed into it——That's horrible.

---

I am sorry to mention this question again, because the problem is still NOT solved even after I make up the other 'half' spectrum as suggestted by @flinty.

considering the following code:

n = 1000;
spectrum = Range[n]^-3;
datainv3 = InverseFourier[Join[spectrum, Reverse@spectrum]];
Show[
LogLogPlot[x^-3, {x, 1, 2 n}, PlotStyle -> {Gray,Dashed}, 
Filling -> Bottom], 
ListLogLogPlot[Abs[Fourier[datainv3[[;; 2 n - 1]]]], PlotStyle -> Red],
ListLogLogPlot[Re[Fourier[datainv3[[;; 2 n - 1]]]], PlotStyle -> Green]]

It gives this:

It is, emmm, I have to say, not much better than the original result.

It is a bit better if I Re datainv3 before taking the Fourier. The code reads:

n = 1000;
spectrum = Range[n]^-3;
datainv3 = InverseFourier[Join[spectrum, Reverse@spectrum]];
Show[
LogLogPlot[x^-3, {x, 1, 2 n}, PlotStyle -> {Gray, Dashed}, Filling -> Bottom], 
ListLogLogPlot[Abs[Fourier[Re@datainv3[[;; 2 n - 1]]]], PlotStyle -> Red],
ListLogLogPlot[Re[Fourier[Re@datainv3[[;; 2 n - 1]]]], PlotStyle -> Green],
ListLogLogPlot[Re[Fourier[Re@datainv3[[;; 2 n - 10]]]], PlotStyle -> Pink]
]

It gives me this

It looks better than the above picture but can hardly be satisfying. datainv3 is 2000 numbers long. However, the Fourier spectrum varies significantly if I miss out on only 10 data points. Does this mean that the Fourier spectrums cannot be compared if they are from original data series with different lengths?

I mean, in my understanding, the spectrum of a signal is its intrinsic properties. The spectrum should not vary greatly if we analyze examples with different lengths. For example, the turbulent kinetic energy spectrum is a -5/3 power law (http://brennen.caltech.edu/fluidbook/basicfluiddynamics/turbulence/turbulencescales.pdf). It is a -5/3 power law if you observe turbulence for 10 minutes, and it is a -5/3 power law if you observe turbulence for 1 hour. Perhaps the low-frequency part may vary a bit as the sign length changes (longer signal length may introduce the lower frequency part), but the high-frequency part should remain the same because the latter represents the small-scale fluctuation.

Answer 1

I think we are now getting to the true question the OP is asking. The question seems to be why is it if I chop a little bit of the end of my signal it makes such a big difference?

I am assuming that the signal is in the time domain and we have been given a representation in the frequency domain. The chopping is done in the time domain. We then compare the Fourier transform of the un-chopped and chopped signal in the frequency domain. OP please confirm that this the question you are asking?

Let's make the signal in the frequency domain and then get the time domain version; we have.

    a = #^(-3) & /@ Range@1000;
    b = Join[a, Reverse[Rest[a]]];
    c = InverseFourier[b];
 
ListLogLogPlot[a, PlotHighlighting -> None, PlotLabel -> "Frequency domain"]
ListLinePlot[c, PlotHighlighting -> None, PlotLabel -> "Time domain"]

Now chop the time domain, and go back to the frequency domain and compare

 c1 = c[[1 ;; -10]];
b1 = Fourier[c1];
ListLogLogPlot[Abs[{b, b1}], PlotHighlighting -> None, 
 Joined -> True,
 PlotLegends -> LineLegend[{"Un-chopped", "Chopped"}]]

So just removing 10 points in the time domain has made a huge difference in the frequency domain.

A point to note is that the data in the time domain is large at the end of the data. This means that Fourier is calculating a Fourier series under the assumption that the data is periodic. When the signal is chopped there is a jump between the end of one period and the start of the next. We can see this if we make the time data periodic. Thus we make two periods from the time data and look at the join between the periods.

c2 = Join[c, c];
c3 = Join[c1, c1];
ListLinePlot[{c2[[2000 - 20 ;; 2000 + 20]], 
  c3[[2000 - 20 ;; 2000 + 20]]}, PlotHighlighting -> None,
 PlotLegends -> LineLegend[{"Un-chopped", "Chopped"}]]

There is a clear jump in the data for the chopped case. This chop is modelled by high frequencies in the frequency domain which is clear from the frequency plot of the chopped data.

So the OP is asking how do we put the chopped data back. One approach could be to interpolate the missing data. I will have to think about this and come back

Answer 2

It's because your initial data that you pass into the inverse transform is missing the other 'half' before you invert it.

One annoying property of Fourier that continually trips people up is that it produces a symmetric spectrum when applied to real valued functions/data. Before you do the inverse, you are missing the other half. This is why the result of Fourier[InverseFourier[...]] contains unexpected imaginary stuff and presumably why you took the Abs. If you add the right hand part of the spectrum then it works, and the only imaginary stuff you'll see will be due to numerical error, hence why I'm taking the Re part.

n = 1000;
spectrum = Range[n]^-3;
datainv3 = InverseFourier[Join[spectrum, Reverse@Rest@spectrum]];
Show[LogLogPlot[x^-3, {x, 1, n}, PlotStyle -> {Gray, Dashed}, 
  Filling -> Bottom], 
 ListLogLogPlot[Re[Fourier[datainv3][[;; n]]], PlotStyle -> Red]]

Answer 3

Note that supplying the reverse is along the correct lines but not quite right. One point has to be dropped. Below I make the data call it a. Then I just join on the reverse which is b. I also drop one point and then join on the reversed data call it b1. Finally I plot the last 10 points of each list

a = #^(-3) & /@ Range@1000;
b = Join[a, Reverse[a]];
b1 = Join[a, Reverse[Rest[a]]]; ListLinePlot[{b[[-10 ;; -1]], 
  b1[[-10 ;; -1]]}, PlotHighlighting -> None, PlotRange -> All]
 

Note that b returns to the first point at the end and b1 does not, it has one less point. Now we do the inverse Fourier transform and plot both the real and imaginary parts of the results

inv = InverseFourier[N[b]];
inv1 = InverseFourier[N[b1]];
ListPlot[Re@{inv, inv1}, PlotHighlighting -> None]
ListPlot[Im@{inv, inv1}, PlotHighlighting -> None]

Note that without and with the dropped point have different real parts and that the imaginary part without the dropped point is significant. With the dropped point the imaginary part is negligible noise.

What is happening here? If we are going from the time domain to the frequency domain via Fourier then we have equally sampled points in the time domain and this results in a periodic function in the frequency domain. Also, the Fourier transform provides both positive and negative frequencies. Fourier calculates just one period. By tradition it does the positive frequencies first and then the negative frequencies -periodic so correct. Each positive frequency has a corresponding negative frequency except for the zero frequency. The zero frequency value does not have a corresponding negative value. Thus when defining data in the frequency domain the first point is the zero frequency - which must not be repeated. As the test function is very large for the zero frequency the effect of repeating it wrongly is significant and shows up.

Hope that helps.