5
$\begingroup$

How can I create function, that will not unpack packed arrays with Map/Table. For example I want read bytes from opened stream (this is my original goal):

getBytesFromStream[stream_, pos_, len_, converter_] := 
Module[{bytes}, 
    SetStreamPosition[stream, pos]; 
    bytes = BinaryReadList[stream, "Byte", len]; 
    converter[bytes]
]

bytes = 
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes}, 
    bytes = getBytesFromStream[stream, 0, 128, #&];
    Close[stream];
    bytes
];

bytesList = 
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes}, 
    bytes = Table[getBytesFromStream[stream, i, 128, #&], {i, 0, 127, 128}];
    Close[stream];
    bytes
];

Developer`PackedArrayQ[bytes]
Developer`PackedArrayQ[bytesList]

True
False

Table did unpack array that was be read. I can pack it again. But if I have only 1Gb free memory and if I want to read file that has size about 400~500Mb - Mathematica can not process this. Unpacked 512Mb in the Mathematica memory has size about 1536Mb. Therefore, I want the function to return the packed result.

$\endgroup$
  • 1
    $\begingroup$ Do you have a (cooked-down!) example file by chance? It will be easier and less error prone to deal with real data sets. $\endgroup$ – Henrik Schumacher Jul 23 '18 at 10:15
  • 1
    $\begingroup$ There are no mistakes here. In the file just zeros. I'm wondering why the function Table (and Map /..) unpacks the list, which is originally packed. $\endgroup$ – Kirill Belov Jul 23 '18 at 10:17
  • 1
    $\begingroup$ Ah. So would make a better post by just ranting about Table[ConstantArray[0., 3], {10}] // Developer`PackedArrayQ and Identity /@ ConstantArray[0., {10, 3}] // Developer`PackedArrayQ. This stream business is actually not related to the question. Agreed, this is a very annoying issue. $\endgroup$ – Henrik Schumacher Jul 23 '18 at 10:25
  • $\begingroup$ Do you know in advance what size the final array will be? In that case, preallocating (e.g. with ConstantArray) and writing into it should resolve the issue. If not, loading smaller chucks into preallocated arrays as Joining them afterwards should also work fine. $\endgroup$ – Henrik Schumacher Jul 23 '18 at 10:27
  • $\begingroup$ @HenrikSchumacher you are right! I can create ConstantArray before reading from the file. I know file size previously! $\endgroup$ – Kirill Belov Jul 23 '18 at 12:00
6
$\begingroup$

If I follow your example, and Henrik's comment, I think there is a misunderstanding here. Table does not unpack; the elements themselves remain packed arrays:

tab = Table[ConstantArray[0., 3], {10}];

Developer`PackedArrayQ /@ tab
{True, True, True, True, True}

Note that packing this outer level saves very little space:

tab2 = Developer`ToPackedArray[tab];

ByteCount[tab]
ByteCount[tab2]
41600

41080

Note also that for large table, specifically exceeding the size given in SystemOptions["CompileOptions" -> "TableCompileLength"] the outer level is packed:

 Table[ConstantArray[0., 5], {250}] // Developer`PackedArrayQ
True

Likewise Map does not fully unpack the array, or a command like Developer`PackedArrayQ /@ tab would return {False, False, ...}.


Update and clarification

Michael E2 pointed out in a comment that Table only returns a packed array if the iterated expression can be compiled. This is important and I was remiss to ignore it, especially as it relates to your question. Indeed standard definitions such as f[x_] := . . . are not compiled and Table does not return an externally packed array. (If the elements returned by such a function are packed they are not unpacked, as explained above.)

f[_] := ConstantArray[0., 5]

tab3 = Table[f[i], {i, 250}];

tab3 // Developer`PackedArrayQ

And @@ Developer`PackedArrayQ /@ tab3
False

True

In this specifically chosen example where the number of elements (250) is much larger than the size of each element the size difference between this output and a fully packed array is significant:

tab3 // ByteCount

tab3 // Developer`ToPackedArray // ByteCount
38056

10360

If this is representative of your application you may wish to operate on the data in blocks, packing each block as you go; something like this:

tab4 =
  Join @@ Table[
    Developer`ToPackedArray @ 
      Table[f[i + 50 j], {i, 50}],
    {j, 0, 4}
  ];

tab4 // Developer`PackedArrayQ

tab4 // ByteCount
True

10360
$\endgroup$
  • 3
    $\begingroup$ Note that the output of Table[] being packed depends on the expression being compilable. (In the OP's case, I think it's not compilable, but it's not easy to test it.) $\endgroup$ – Michael E2 Jul 23 '18 at 14:07
  • $\begingroup$ @MichaelE2 Please see my update and tell me if I have adequately addressed this. $\endgroup$ – Mr.Wizard Jul 24 '18 at 9:04
  • $\begingroup$ Yes, I believe it does address it as well as can be. I suggested ReadByteArray in a comment above, which I think has the potential to be a better solution in the OP's actual use-case. However, there's not enough information (esp. about the OP's function converter[]), nor do I have enough experience with ByteArray objects, to make a confident recommendation. $\endgroup$ – Michael E2 Jul 24 '18 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.