# Why does Rescale performance vary significantly based on the real number values in a list?

I have a list of the form:

testList = Table[{Flatten[RandomReal[{-200, 200}, {1, 2}], 1], RandomReal[{-200, 200}, {69, 2}]}, {i, 1, 100}];


And I would like to perform the following rescaling operation very quickly:

Rescale[testList, {-160 Sqrt[5], 160 Sqrt[2]}] // AbsoluteTiming


And indeed we see that this takes about $\approx 1$ millisecond.

However, if we manually specify testList:

riggedTestList = {{72.0003, 254.632}, {{0., 0.}, {164.523, -1.48019}, {-36.4013, -1.6529}, {-30.6006, -9.68836}, {79.1769, -9.38496}, {127.414, -13.3351}, {26.2267, -14.9839}, {101.471, -17.2131}, {112.759, -20.2044}, {140.786, -20.3125}, {152.117, -23.4485}, {171.028, -28.6424}, {128.855, -34.8548}, {67.2216, -48.2581}, {-58.5145, -53.227}, {147.275, -52.8399}, {-2.11577, -54.7374}, {160.367, -57.3066}, {-43.8697, -59.9146}, {50.3993, -63.4993}, {105.478, -64.6626}, {-38.5779, -73.1765}, {167.692, -74.2231}, {38.1779, -75.0274}, {182.719, -75.3023}, {101.273, -76.3709}, {91.2674, -78.7477}, {140.126, -83.058}, {167.292, -83.068}, {-30.6985, -90.7144}, {73.0603, -91.5605}, {15.3827, -93.4644}, {168.049, -97.4066}, {101.349, -97.9227}, {148.875, -105.038}, {-27.7536, -106.1}, {119.991, -110.415}, {170.865, -110.703}, {-1.03466, -111.89}, {96.2519, -121.383}, {179.463, -127.087}, {-62.9256, -127.913}, {172.077, -131.731}, {-41.8538, -137.851}, {171.768, -143.748}, {28.4562, -146.02}, {71.0586, -145.51}, {149.767, -146.569}, {37.6894, -150.356}, {-26.1955, -159.44}, {-41.9501, -163.069}, {135.019, -164.213}, {-62.5847, -172.048}, {7.02217, -174.589}, {133.175, -175.744}, {0.392205, -184.99}, {56.418, -189.734}, {83.9584, -196.068}, {-9.36728, -198.523}, {141.564, -197.764}, {43.6112, -200.002}, {133.577, -201.431}, {16.5308, -203.89}, {79.8118, -207.938}, {178.305, -210.113}, {-45.2895, -216.227}, {-4.60035, -234.998}, {3.16025, -240.952}, {78.6896, -248.948}}};

riggedTestList = Table[riggedTestList, {i, 1, 100}];


And then attempt the same rescale on this array with the same dimensions as testList:

Rescale[riggedTestList, {-160 Sqrt[5], 160 Sqrt[2]}] // AbsoluteTiming


The Rescale operation takes $\approx 33$ milliseconds, a difference of over an order of magnitude. How could this be the case? Both DeveloperPackedArrayQ[testList] and DeveloperPackedArrayQ[riggedTestList] return False, so I don't think it's a list packing issue?

-

Actually, it is all about packing. By using RandomReal you an generating packed sub-arrays even if the complete array is not packed (and can't be, due to irregular shape):

Map[PackedArrayQ, testList, {2}] // Short

{{True,True},{True,True},{True,True},{True,True},{True,True},<<91>>,{True,True},
{True,True},{True,True},{True,True}}


Let's look at some timings more closely:

SetAttributes[timeAvg, HoldFirst]
timeAvg[func_] := Do[If[# > 0.3, Return[#/5^i]] & @@ Timing@Do[func, {5^i}], {i, 0, 15}]

Needs["Developer"]

Rescale[testList, {-160 Sqrt[5], 160 Sqrt[2]}] // timeAvg

0.0005744

Rescale[riggedTestList, {-160 Sqrt[5], 160 Sqrt[2]}] // timeAvg

0.0256


Fully unpacking testList results in the same timing as riggedTestList:

unpacked = FromPackedArray @ testList;
Rescale[unpacked, {-160 Sqrt[5], 160 Sqrt[2]}] // timeAvg

0.0256


Packing riggedTestList in the same degree as testList also matches the faster timing:

repacked = Map[ToPackedArray, riggedTestList, {2}];
Rescale[repacked, {-160 Sqrt[5], 160 Sqrt[2]}] // timeAvg

0.0005744


Addressing a comment below, in version 7 under Windows the results are equivalent as expected:

Rescale[testList, {-160 Sqrt[5], 160 Sqrt[2]}] ==
Rescale[unpacked, {-160 Sqrt[5], 160 Sqrt[2]}]

Rescale[riggedTestList, {-160 Sqrt[5], 160 Sqrt[2]}] ==
Rescale[repacked, {-160 Sqrt[5], 160 Sqrt[2]}]
`

True

True

-
Ahh, I should have looked at the sublists. – RM1618 Sep 23 '13 at 7:56
@RM1618 These things are fairly tricky. I didn't understand this stuff until someone else showed me. – Mr.Wizard Sep 23 '13 at 7:57
Well thanks for passing along the knowledge. – RM1618 Sep 23 '13 at 8:01
@RM1618 For that you're always welcome. :-) – Mr.Wizard Sep 23 '13 at 8:02
Is it possible to have repacked actually look like Rescale[riggedTestList, {-160 Sqrt[5], 160 Sqrt[2]}] after rescaling? For me on version 9, I'm not seeing the Rescale operation performed? – RM1618 Sep 23 '13 at 8:10