Packing and unpacking of arrays with Map/Table

Question

How can I create function, that will not unpack packed arrays with Map/Table. For example I want read bytes from opened stream (this is my original goal):

getBytesFromStream[stream_, pos_, len_, converter_] := 
Module[{bytes}, 
    SetStreamPosition[stream, pos]; 
    bytes = BinaryReadList[stream, "Byte", len]; 
    converter[bytes]
]

bytes = 
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes}, 
    bytes = getBytesFromStream[stream, 0, 128, #&];
    Close[stream];
    bytes
];

bytesList = 
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes}, 
    bytes = Table[getBytesFromStream[stream, i, 128, #&], {i, 0, 127, 128}];
    Close[stream];
    bytes
];

Developer`PackedArrayQ[bytes]
Developer`PackedArrayQ[bytesList]

True
False

Table did unpack array that was be read. I can pack it again. But if I have only 1Gb free memory and if I want to read file that has size about 400~500Mb - Mathematica can not process this. Unpacked 512Mb in the Mathematica memory has size about 1536Mb. Therefore, I want the function to return the packed result.

Answer 1

If I follow your example, and Henrik's comment, I think there is a misunderstanding here. Table does not unpack; the elements themselves remain packed arrays:

tab = Table[ConstantArray[0., 3], {10}];

Developer`PackedArrayQ /@ tab

{True, True, True, True, True}

Note that packing this outer level saves very little space:

tab2 = Developer`ToPackedArray[tab];

ByteCount[tab]
ByteCount[tab2]

41600

41080

Note also that for large table, specifically exceeding the size given in SystemOptions["CompileOptions" -> "TableCompileLength"] the outer level is packed:

 Table[ConstantArray[0., 5], {250}] // Developer`PackedArrayQ

True

Likewise Map does not fully unpack the array, or a command like Developer`PackedArrayQ /@ tab would return {False, False, ...}.

---

Update and clarification

Michael E2 pointed out in a comment that Table only returns a packed array if the iterated expression can be compiled. This is important and I was remiss to ignore it, especially as it relates to your question. Indeed standard definitions such as f[x_] := . . . are not compiled and Table does not return an externally packed array. (If the elements returned by such a function are packed they are not unpacked, as explained above.)

f[_] := ConstantArray[0., 5]

tab3 = Table[f[i], {i, 250}];

tab3 // Developer`PackedArrayQ

And @@ Developer`PackedArrayQ /@ tab3

False

True

In this specifically chosen example where the number of elements (250) is much larger than the size of each element the size difference between this output and a fully packed array is significant:

tab3 // ByteCount

tab3 // Developer`ToPackedArray // ByteCount

38056

10360

If this is representative of your application you may wish to operate on the data in blocks, packing each block as you go; something like this:

tab4 =
  Join @@ Table[
    Developer`ToPackedArray @ 
      Table[f[i + 50 j], {i, 50}],
    {j, 0, 4}
  ];

tab4 // Developer`PackedArrayQ

tab4 // ByteCount

True

10360