# Packing and unpacking of arrays with Map/Table

How can I create function, that will not unpack packed arrays with Map/Table. For example I want read bytes from opened stream (this is my original goal):

getBytesFromStream[stream_, pos_, len_, converter_] :=
Module[{bytes},
SetStreamPosition[stream, pos];
bytes = BinaryReadList[stream, "Byte", len];
converter[bytes]
]

bytes =
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes},
bytes = getBytesFromStream[stream, 0, 128, #&];
Close[stream];
bytes
];

bytesList =
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes},
bytes = Table[getBytesFromStream[stream, i, 128, #&], {i, 0, 127, 128}];
Close[stream];
bytes
];

DeveloperPackedArrayQ[bytes]
DeveloperPackedArrayQ[bytesList]


True
False

Table did unpack array that was be read. I can pack it again. But if I have only 1Gb free memory and if I want to read file that has size about 400~500Mb - Mathematica can not process this. Unpacked 512Mb in the Mathematica memory has size about 1536Mb. Therefore, I want the function to return the packed result.

• Do you have a (cooked-down!) example file by chance? It will be easier and less error prone to deal with real data sets. Jul 23, 2018 at 10:15
• There are no mistakes here. In the file just zeros. I'm wondering why the function Table (and Map /..) unpacks the list, which is originally packed. Jul 23, 2018 at 10:17
• Ah. So would make a better post by just ranting about Table[ConstantArray[0., 3], {10}] // DeveloperPackedArrayQ and Identity /@ ConstantArray[0., {10, 3}] // DeveloperPackedArrayQ. This stream business is actually not related to the question. Agreed, this is a very annoying issue. Jul 23, 2018 at 10:25
• Do you know in advance what size the final array will be? In that case, preallocating (e.g. with ConstantArray) and writing into it should resolve the issue. If not, loading smaller chucks into preallocated arrays as Joining them afterwards should also work fine. Jul 23, 2018 at 10:27
• @HenrikSchumacher you are right! I can create ConstantArray before reading from the file. I know file size previously! Jul 23, 2018 at 12:00

## 1 Answer

If I follow your example, and Henrik's comment, I think there is a misunderstanding here. Table does not unpack; the elements themselves remain packed arrays:

tab = Table[ConstantArray[0., 3], {10}];

DeveloperPackedArrayQ /@ tab

{True, True, True, True, True}


Note that packing this outer level saves very little space:

tab2 = DeveloperToPackedArray[tab];

ByteCount[tab]
ByteCount[tab2]

41600

41080


Note also that for large table, specifically exceeding the size given in SystemOptions["CompileOptions" -> "TableCompileLength"] the outer level is packed:

 Table[ConstantArray[0., 5], {250}] // DeveloperPackedArrayQ

True


Likewise Map does not fully unpack the array, or a command like DeveloperPackedArrayQ /@ tab would return {False, False, ...}.

## Update and clarification

Michael E2 pointed out in a comment that Table only returns a packed array if the iterated expression can be compiled. This is important and I was remiss to ignore it, especially as it relates to your question. Indeed standard definitions such as f[x_] := . . . are not compiled and Table does not return an externally packed array. (If the elements returned by such a function are packed they are not unpacked, as explained above.)

f[_] := ConstantArray[0., 5]

tab3 = Table[f[i], {i, 250}];

tab3 // DeveloperPackedArrayQ

And @@ DeveloperPackedArrayQ /@ tab3

False

True


In this specifically chosen example where the number of elements (250) is much larger than the size of each element the size difference between this output and a fully packed array is significant:

tab3 // ByteCount

tab3 // DeveloperToPackedArray // ByteCount

38056

10360


If this is representative of your application you may wish to operate on the data in blocks, packing each block as you go; something like this:

tab4 =
Join @@ Table[
DeveloperToPackedArray @
Table[f[i + 50 j], {i, 50}],
{j, 0, 4}
];

tab4 // DeveloperPackedArrayQ

tab4 // ByteCount

True

10360

• Note that the output of Table[] being packed depends on the expression being compilable. (In the OP's case, I think it's not compilable, but it's not easy to test it.) Jul 23, 2018 at 14:07
• @MichaelE2 Please see my update and tell me if I have adequately addressed this. Jul 24, 2018 at 9:04
• Yes, I believe it does address it as well as can be. I suggested ReadByteArray in a comment above, which I think has the potential to be a better solution in the OP's actual use-case. However, there's not enough information (esp. about the OP's function converter[]), nor do I have enough experience with ByteArray` objects, to make a confident recommendation. Jul 24, 2018 at 22:47