Calculation time differs between segmented loops and one whole loop

I stumbled on a strange issue while simulating a large number of random values, $$10^8$$ to be precise.

I generated the numbers at first by the following code:

\[Alpha]=0.2;
levytab2 = {};
While[Length[levytab2] < 10^8,
Clear[u1, u2];
u1 = RandomVariate[UniformDistribution[{0, 1}]];
u2 = RandomVariate[UniformDistribution[{0, 1}]];
AppendTo[
levytab2, (1/Abs[Log[u1]]*(
Sin[(1 - \[Alpha])*Pi*u2]*(Sin[\[Alpha]*Pi*u2])^(\[Alpha]/(
1 - \[Alpha])))/(Sin[Pi*u2])^(1/(1 - \[Alpha])))^((
1 - \[Alpha])/\[Alpha])]
];


This took two days to calculate up to 6 000 000 numbers.. this is riddiculous. I tested the code with sizes of $$10^3$$, $$10^4$$ and $$10^5$$ and found that their calculation time does not scale by 10 as the sizes do. I then tried to just do a smaller sample size 10 times. This indeed produced calculation times scaling by 10, like the sample size.

For example, simulating $$10^4$$ numbers took 0.390625 seconds. Simulating $$10^5$$ numbers took 18.62500 seconds. This is not 10*time-to-simulate-$$10^4$$ numbers. However, simulating 10 times the $$10^4$$ points took 3.56 seconds - much more in lines with what is expected.

I then simulated $$10^8$$ numbers by nesting Do loops of size 10, in the spirit of:

Do[
levytabcbind2 = {};
Do[
levytab2 = {};
While[Length[levytab2] < 10^4,
Clear[u1, u2];
u1 = RandomVariate[UniformDistribution[{0, 1}]];
u2 = RandomVariate[UniformDistribution[{0, 1}]];
AppendTo[
levytab2, (1/Abs[Log[u1]]*(
Sin[(1 - \[Alpha])*Pi*u2]*(Sin[\[Alpha]*Pi*u2])^(\[Alpha]/(
1 - \[Alpha])))/(Sin[Pi*u2])^(1/(1 - \[Alpha])))^((
1 - \[Alpha])/\[Alpha])]
];
Clear[dummy];
dummy = i;
AppendTo[levytabcbind, levytab2], {i, 1, 10}]
Clear[dummy2];
dummy2 = j;
AppendTo[levytabcbind2, levytabcbind], {j, 1, 10}];


In the end, it took an hour to simulate the $$10^8$$ points, as the base While loop takes only about 3.5 seconds.

My question is: why does Mathematica take exponentially longer to work through a Do loop of larger size, while doing the same number of calculations in smaller nested Do loops takes the expected time, given sample size growing by 10?

• Because you are repeatedly doing AppendTo, the worst way of accumulating lists in Mathematica. If you want a list of things, use Table. Commented Apr 30, 2020 at 12:25
• Aha, but do you know of a simpe way of explaining why the blow-up of calculation time? Or is it too technical? Also: Could Mathematica handle better then a Table[] command of the same variate? Im wondering how to reset the random variates u1 and u2 inside of it... Commented Apr 30, 2020 at 12:29
• Yes, the reason is that Mathematica cannot predict what you are going to add to the list next; a datum of the same type as the ones already there (ints, doubles, chars) or something completely different like a Word document, PDF, or an image. Hence, it must reallocate on each append, forcing a copy of the existing list. This is scales horribly because each time you append, a longer and longer list must be copied. Commented Apr 30, 2020 at 12:31
• Generate all the random variates right from the start: u1 = RandomVariate[UniformDistribution[{0,1}], 10^6] and you have one million values, and similarly for u2. Then simply apply your function to these lists, because all your operations (Sin, Log etc.) are listable, so they will thread over u1 and u2 automatically. Commented Apr 30, 2020 at 12:34

u1 = RandomVariate[UniformDistribution[{0, 1}], 10^8];

• It was like that for me when I started using Mathematica, coming from compiled languages where AppendTo-like things tend to be constant-time. A good rule of thumb in Mathematica is to use as few loops as possible (For, Do, While etc.) Commented Apr 30, 2020 at 12:52