0
$\begingroup$

I stumbled on a strange issue while simulating a large number of random values, $10^8$ to be precise.

I generated the numbers at first by the following code:

\[Alpha]=0.2; 
levytab2 = {};
While[Length[levytab2] < 10^8,
 Clear[u1, u2];
 u1 = RandomVariate[UniformDistribution[{0, 1}]];
 u2 = RandomVariate[UniformDistribution[{0, 1}]];
 AppendTo[
  levytab2, (1/Abs[Log[u1]]*(
    Sin[(1 - \[Alpha])*Pi*u2]*(Sin[\[Alpha]*Pi*u2])^(\[Alpha]/(
     1 - \[Alpha])))/(Sin[Pi*u2])^(1/(1 - \[Alpha])))^((
  1 - \[Alpha])/\[Alpha])]
 ];

This took two days to calculate up to 6 000 000 numbers.. this is riddiculous. I tested the code with sizes of $10^3$, $10^4$ and $10^5$ and found that their calculation time does not scale by 10 as the sizes do. I then tried to just do a smaller sample size 10 times. This indeed produced calculation times scaling by 10, like the sample size.

For example, simulating $10^4$ numbers took 0.390625 seconds. Simulating $10^5$ numbers took 18.62500 seconds. This is not 10*time-to-simulate-$10^4$ numbers. However, simulating 10 times the $10^4$ points took 3.56 seconds - much more in lines with what is expected.

I then simulated $10^8$ numbers by nesting Do loops of size 10, in the spirit of:

Do[
  levytabcbind2 = {};
  Do[
    levytab2 = {};
    While[Length[levytab2] < 10^4,
     Clear[u1, u2];
     u1 = RandomVariate[UniformDistribution[{0, 1}]];
     u2 = RandomVariate[UniformDistribution[{0, 1}]];
     AppendTo[
      levytab2, (1/Abs[Log[u1]]*(
        Sin[(1 - \[Alpha])*Pi*u2]*(Sin[\[Alpha]*Pi*u2])^(\[Alpha]/(
         1 - \[Alpha])))/(Sin[Pi*u2])^(1/(1 - \[Alpha])))^((
      1 - \[Alpha])/\[Alpha])]
     ];
    Clear[dummy];
    dummy = i;
    AppendTo[levytabcbind, levytab2], {i, 1, 10}]
   Clear[dummy2];
  dummy2 = j;
  AppendTo[levytabcbind2, levytabcbind], {j, 1, 10}];

In the end, it took an hour to simulate the $10^8$ points, as the base While loop takes only about 3.5 seconds.

My question is: why does Mathematica take exponentially longer to work through a Do loop of larger size, while doing the same number of calculations in smaller nested Do loops takes the expected time, given sample size growing by 10?

$\endgroup$
4
  • $\begingroup$ Because you are repeatedly doing AppendTo, the worst way of accumulating lists in Mathematica. If you want a list of things, use Table. $\endgroup$ – Marius Ladegård Meyer Apr 30 '20 at 12:25
  • $\begingroup$ Aha, but do you know of a simpe way of explaining why the blow-up of calculation time? Or is it too technical? Also: Could Mathematica handle better then a Table[] command of the same variate? Im wondering how to reset the random variates u1 and u2 inside of it... $\endgroup$ – Castle Apr 30 '20 at 12:29
  • 1
    $\begingroup$ Yes, the reason is that Mathematica cannot predict what you are going to add to the list next; a datum of the same type as the ones already there (ints, doubles, chars) or something completely different like a Word document, PDF, or an image. Hence, it must reallocate on each append, forcing a copy of the existing list. This is scales horribly because each time you append, a longer and longer list must be copied. $\endgroup$ – Marius Ladegård Meyer Apr 30 '20 at 12:31
  • 1
    $\begingroup$ Generate all the random variates right from the start: u1 = RandomVariate[UniformDistribution[{0,1}], 10^6] and you have one million values, and similarly for u2. Then simply apply your function to these lists, because all your operations (Sin, Log etc.) are listable, so they will thread over u1 and u2 automatically. $\endgroup$ – Marius Ladegård Meyer Apr 30 '20 at 12:34
2
$\begingroup$

Do this:

u1 = RandomVariate[UniformDistribution[{0, 1}], 10^8];
u2 = RandomVariate[UniformDistribution[{0, 1}], 10^8];
alpha = 0.2;
ans = (1/Abs[Log[u1]]*(Sin[(1 - alpha)*Pi*u2]*(Sin[alpha*Pi*u2])^(alpha/(1 - alpha)))/(Sin[Pi*u2])^(1/(1 - alpha)))^((1 - alpha)/alpha);

The whole thing takes 23 seconds on my computer.

$\endgroup$
2
  • $\begingroup$ This makes me feel very dumb and thankful. Luckly my calculatons already work, but thanks for the insight! Will be much more effective on the next runs. Thank you $\endgroup$ – Castle Apr 30 '20 at 12:45
  • 1
    $\begingroup$ It was like that for me when I started using Mathematica, coming from compiled languages where AppendTo-like things tend to be constant-time. A good rule of thumb in Mathematica is to use as few loops as possible (For, Do, While etc.) $\endgroup$ – Marius Ladegård Meyer Apr 30 '20 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.