7
$\begingroup$

How can I create function, that will not unpack packed arrays with Map/Table. For example I want read bytes from opened stream (this is my original goal):

getBytesFromStream[stream_, pos_, len_, converter_] := 
Module[{bytes}, 
    SetStreamPosition[stream, pos]; 
    bytes = BinaryReadList[stream, "Byte", len]; 
    converter[bytes]
]

bytes = 
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes}, 
    bytes = getBytesFromStream[stream, 0, 128, #&];
    Close[stream];
    bytes
];

bytesList = 
Module[{stream = OpenRead["test.dat", BinaryFormat -> True], bytes}, 
    bytes = Table[getBytesFromStream[stream, i, 128, #&], {i, 0, 127, 128}];
    Close[stream];
    bytes
];

Developer`PackedArrayQ[bytes]
Developer`PackedArrayQ[bytesList]

True
False

Table did unpack array that was be read. I can pack it again. But if I have only 1Gb free memory and if I want to read file that has size about 400~500Mb - Mathematica can not process this. Unpacked 512Mb in the Mathematica memory has size about 1536Mb. Therefore, I want the function to return the packed result.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Do you have a (cooked-down!) example file by chance? It will be easier and less error prone to deal with real data sets. $\endgroup$
    – Henrik Schumacher
    Jul 23, 2018 at 10:15
  • 1
    $\begingroup$ There are no mistakes here. In the file just zeros. I'm wondering why the function Table (and Map /..) unpacks the list, which is originally packed. $\endgroup$
    – Kirill Belov
    Jul 23, 2018 at 10:17
  • 1
    $\begingroup$ Ah. So would make a better post by just ranting about Table[ConstantArray[0., 3], {10}] // Developer`PackedArrayQ and Identity /@ ConstantArray[0., {10, 3}] // Developer`PackedArrayQ. This stream business is actually not related to the question. Agreed, this is a very annoying issue. $\endgroup$
    – Henrik Schumacher
    Jul 23, 2018 at 10:25
  • $\begingroup$ Do you know in advance what size the final array will be? In that case, preallocating (e.g. with ConstantArray) and writing into it should resolve the issue. If not, loading smaller chucks into preallocated arrays as Joining them afterwards should also work fine. $\endgroup$
    – Henrik Schumacher
    Jul 23, 2018 at 10:27
  • $\begingroup$ @HenrikSchumacher you are right! I can create ConstantArray before reading from the file. I know file size previously! $\endgroup$
    – Kirill Belov
    Jul 23, 2018 at 12:00

1 Answer 1

Reset to default
8
$\begingroup$

If I follow your example, and Henrik's comment, I think there is a misunderstanding here. Table does not unpack; the elements themselves remain packed arrays:

tab = Table[ConstantArray[0., 3], {10}];

Developer`PackedArrayQ /@ tab

{True, True, True, True, True}

Note that packing this outer level saves very little space:

tab2 = Developer`ToPackedArray[tab];

ByteCount[tab]
ByteCount[tab2]

41600

41080

Note also that for large table, specifically exceeding the size given in SystemOptions["CompileOptions" -> "TableCompileLength"] the outer level is packed:

 Table[ConstantArray[0., 5], {250}] // Developer`PackedArrayQ

True

Likewise Map does not fully unpack the array, or a command like Developer`PackedArrayQ /@ tab would return {False, False, ...}.

Update and clarification

Michael E2 pointed out in a comment that Table only returns a packed array if the iterated expression can be compiled. This is important and I was remiss to ignore it, especially as it relates to your question. Indeed standard definitions such as f[x_] := . . . are not compiled and Table does not return an externally packed array. (If the elements returned by such a function are packed they are not unpacked, as explained above.)

f[_] := ConstantArray[0., 5]

tab3 = Table[f[i], {i, 250}];

tab3 // Developer`PackedArrayQ

And @@ Developer`PackedArrayQ /@ tab3

False

True

In this specifically chosen example where the number of elements (250) is much larger than the size of each element the size difference between this output and a fully packed array is significant:

tab3 // ByteCount

tab3 // Developer`ToPackedArray // ByteCount

38056

10360

If this is representative of your application you may wish to operate on the data in blocks, packing each block as you go; something like this:

tab4 =
  Join @@ Table[
    Developer`ToPackedArray @ 
      Table[f[i + 50 j], {i, 50}],
    {j, 0, 4}
  ];

tab4 // Developer`PackedArrayQ

tab4 // ByteCount

True

10360
Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ Note that the output of Table[] being packed depends on the expression being compilable. (In the OP's case, I think it's not compilable, but it's not easy to test it.) $\endgroup$
    – Michael E2
    Jul 23, 2018 at 14:07
  • $\begingroup$ @MichaelE2 Please see my update and tell me if I have adequately addressed this. $\endgroup$
    – Mr.Wizard
    Jul 24, 2018 at 9:04
  • $\begingroup$ Yes, I believe it does address it as well as can be. I suggested ReadByteArray in a comment above, which I think has the potential to be a better solution in the OP's actual use-case. However, there's not enough information (esp. about the OP's function converter[]), nor do I have enough experience with ByteArray objects, to make a confident recommendation. $\endgroup$
    – Michael E2
    Jul 24, 2018 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.