Forcing efficient numerical arrays

Question

I encounter strange behaviour of Mathematica when evaluating multi-dimensional arrays of double-precision data.

Mathematica can express large arrays of machine-precision numbers using a specialized internal data-format, which optimizes memory usage and speeds up operations.

However, Mathematica does not always recognize when it can use such optimized arrays. Below are three cases that would ideally yield the same optimized data structure upon evaluation:

arr1 = Table[1.23, {1}, {100000}];
ByteCount[arr1]
(* 800168 *)

ncols = 1000;
arr2 = Table[1.23, {1}, {ncols}];
ByteCount[arr2]
(* 3200088 *)

ncols = 1000;
arr3 = Table[1.23, {1}, Evaluate[{ncols}]];
ByteCount[arr3]
(* 800168 *)

For some reason, the second case gives sub-optimal behaviour.

My question is this: If I end up with a case where Mathematica does not use its optimized numeric array data structure (such as arr2, above), how can I explicitly force a conversion?

Answer 1

The larger byte count in the second case is because the array is not packed. You can check if an array is packed as:

Developer`PackedArrayQ /@ {arr1, arr2}
(* {True, False} *)

Packed arrays contain only elements of the same kind and so space is saved by not tracking the type of each element. The reason why the first case is packed whereas the second isn't (even though they're essentially the same construct), is that in the second case, the dimensions of the array are not immediately known when Table is evaluated. Table has the attribute HoldAll and receives its iterators in an unevaluated form (necessary for proper scoping). When it sees an integer, it packs as a result of auto-compilation (thanks to Leonid Shifrin for clarifying), which is why Table[..., Evaluate@{ncols}] also returns a packed array in your example.

To convert arr2 to a packed array, use:

arr3 = Developer`ToPackedArray@arr2;
Developer`PackedArrayQ@arr3
(* True *)

You can read more about packed arrays and their advantages in the answers to this question.