# Function to spot randomness and/or predict future data

Does Mathematica have a good function to show if a series of discrete data are random or have some hidden periodicity about them? Am I correct in thinking Fourier analysis is best suited for that. What about predictability of future data points, is there a different function for that?

Here's an example of a data series which I tried to randomly make more predictable:

data = {11, 11, 12, 7, 14, 10, 13, 10, 11, 9, 9, 11, 7, 10, 10, 10, 9,
13, 8, 10, 12, 7, 14, 9, 11, 5, 9, 8, 11, 14, 9, 10, 6, 5, 10, 12,
10, 16, 9, 11, 9, 11, 10, 8, 14, 13, 15, 9, 14, 7, 5, 13, 9, 12, 17,
17, 14, 10, 10, 11, 9, 12, 11, 7, 10, 6, 10, 16, 12, 8, 9, 12, 10,
11, 10, 14, 8, 12, 12, 10, 7, 11, 14, 13, 9, 13, 13, 7, 13, 11, 8,
8, 8, 8, 14, 18, 16, 12, 8, 8, 7, 11, 8, 8, 14, 13, 9, 12, 12, 10,
15, 9, 12, 10, 11, 8, 8, 12, 7, 9, 10, 8, 9, 9, 11, 9, 13, 7, 11, 7,
8, 12, 8, 11, 9, 13, 7, 13, 11, 11, 8, 12, 9, 7, 14, 16, 9, 8, 11,
9, 12, 6, 12, 10, 10, 12, 4, 12, 8, 16, 11, 13, 12, 15, 8, 11, 10,
12, 11, 16, 13, 11, 5, 10, 8, 15, 12, 8, 13, 8, 12, 12, 14, 11, 8,
10, 12, 13, 10, 10, 11, 8, 12, 8, 17, 7, 16, 7, 13, 8, 11, 8, 14, 9,
7, 11, 11, 5, 13, 11, 8, 13, 10, 8, 14, 6, 14, 11, 8, 10, 12, 14,
10, 8, 13, 13, 8, 10, 14, 11, 13, 11, 10, 9, 16, 7, 14, 11, 12, 8,
11, 9, 11, 8, 9, 5, 7, 9, 17, 10, 14, 9, 12, 5, 7, 8, 12, 11, 9, 9,
12, 7, 12, 5, 10, 12, 16, 6, 11, 12, 4, 13, 4, 6, 8, 11, 10, 8, 11,
8, 6, 7, 13, 13, 9, 10, 12, 10, 13, 14, 8, 10, 7, 10, 10, 10, 12, 8,
8, 5, 10, 9, 8}


Then graph it: ListLinePlot[data]

The I try to plot the Fourier as such:

ListLinePlot[Abs[Fourier[data]], DataRange -> {1, 300}]


I notice above that frequencies of roughly 35, 130, 170, and 265 spike up. Does this tell me that the data is periodic around those frequencies? If so it means it will repeat at these periods in the future? I notice if I use more random data, I do not even get nice peak frequencies at all.

Does Mathematica have a better way of doing this? I think I could have used that function in a more efficient way, or are there better ones to use? Am I supposed to use TimeSeriesForecast[], or Predict[], or others? If so, can you show me with example how to do this.

• This is really a question about choosing the appropriate statistical test and would be better asked on stats.SE. Nov 21, 2020 at 8:33

The Wolfram Function Repository has a number of randomness tests which you may find useful:

Let's try SpectralRandomnessTest. We must rescale to [0,1] and make sure the values are reals:

ResourceFunction["SpectralRandomnessTest"][Rescale[N@data], "PValue"]
(* 0.816545 *)


A high $$p$$-value under the KolmogorovSmirnovTest means we cannot reject the null hypothesis that the data are random.

However, the RunLengthRandomnessTest gives us a small $$p$$-value, but not very small:

ResourceFunction["RunLengthRandomnessTest"][Rescale[N@data], "PValue"]
(* 0.0185384 *)


These tests above require extremely low $$p$$-values to draw conclusions. If you test them out on other random noise, the $$p$$-values tend to fluctuate a lot between 0.1 and 0.9, so unless we observe an extremely low $$p$$-value we should continue to assume it is random data.

How about autocorrelation? AutocorrelationTest[data] gives a very lower $$p$$-value of 0.006 suggesting there is probably autocorrelation in the data, and you may want to check visually:

Periodogram[data]
CorrelationFunction[N@data, {0, Length[data] - 1}] // ListLinePlot


I can't say much about TimeSeriesForecast and Predict. They will most likely be completely useless on this data. The best you can do is say it will hover around 10 ±6.

• Thx @flinty. Very useful functions! I tried the first one, for some reason it gives me the p value of 1. So How do I use the result of RunLengthRandomnessTest which I get 0.018, what does that tell me about the next data-points? Same as the AutoCorrelation. I got the graph of that which starts high and reduces towards zero as X->300. Also what does Periodogram do? Thx Nov 21, 2020 at 19:54
• This just provides you some tools to test randomness. There's nothing much you can do to predict the next points reliably. The docs for Periodogram , it's essentially a power spectrum. A very low $p$-value would indicate the data is non-random, but it needs to be very low to draw that conclusion, like < 0.001, (run this multiple times and you'll see why: ResourceFunction["RunLengthRandomnessTest"][RandomReal[1, 100000]]). Nov 21, 2020 at 20:56
• I see, so it's a power spectrum. It shows maximum power of a signal, or the dominant frequency I presume. As an example I ran it for this dataset: $data = Table[Sin[0.5 \[Pi] n], {n, 0, 30}];$ Then when I run Periodogram[data], I get a peak of around 0.26 - does this mean this signal repeats at period of 0.26? ....overall? Nov 21, 2020 at 22:02
• @Steve237 yes, that's about 1/4 Hz. The wave should like like $y = \sin(2 \pi f x)$, so in your case above $2 f=0.5$ Nov 22, 2020 at 14:18