Lets say I create a giant list:
tmp=RandomReal[{1,10},600000000];
Now I want to take 10 random sublists of length 150000:
rand = RandomInteger[{1, 600000000 - 149999}, 10];
First@AbsoluteTiming@Map[Take[tmp, {#, # + 149999}] &, rand]
(*0.007*)
this works fine and is speedy. However if I want to extract a 100 sublists of 150000 I get:
rand = RandomInteger[{1, 600000000 - 149999}, 100];
First@AbsoluteTiming@Map[Take[tmp, {#, # + 149999}] &, rand]
(*87.707*)
An increase of several orders of magnitude in time! If I use Do[]
instead of Map[]
I can speed it up substantially:
foo = Table[{}, 100];
First@AbsoluteTiming@
Do[foo[[i]]=Take[tmp, {rand[[i]], rand[[i]] + 149999}], {i, 1, 100}]
(*0.05*)
Does anyone know what could be causing this?
Max
Map
is trying to compile yourtmp
. Why is there a huge performance gap using Map with more than 100 List entries $\endgroup$RandomReal[{1, 10}, {100, 150000}];
$\endgroup$