13
$\begingroup$

What is the simplest way to partition a list into equal-length sublists, allow the last sublist to be shorter of necessary, and avoid unpacking?

Partition doesn't usually unpack:

arr = RandomReal[1, 100000];

Developer`PackedArrayQ[arr]
(* True *)

Developer`PackedArrayQ /@ Partition[arr, 25000]
(* {True, True, True, True} *)

But it does if we allow the last sublist to be shorter:

Developer`PackedArrayQ /@ Partition[arr, UpTo@30000]
(* {False, False, False, False} *)

Developer`PackedArrayQ /@ Partition[arr, 30000, 30000, {1, 1}, {}]
(* {False, False, False, False} *)

Question 1: Why isn't the result packed? Is there any reason I didn't think of or is this case just not optimized?

Question 2: What's an elegant way to avoid this problem?

$\endgroup$
8
$\begingroup$

One could use Internal`PartitionRagged to get packed subarrays.

Code.

partition[l_List, n_Integer] := With[{qr = QuotientRemainder[Length[l], n]},

   If[qr[[2]] == 0,
      Partition[l, n],
      Internal`PartitionRagged[l, Append[ConstantArray[n, qr[[1]]], qr[[2]]]]
   ]

];

Usage.

list = Range[100];

p1 = partition[list, 25];
Length /@ p1
Developer`PackedArrayQ /@ p1
(* {25, 25, 25, 25} *)
(* {True, True, True, True} *)

p2 = partition[list, 30];
Length /@ p2
Developer`PackedArrayQ /@ p2
(* {30, 30, 30, 10} *)
(* {True, True, True, True} *)
$\endgroup$
  • $\begingroup$ Alternatively, Internal`PartitionRagged[l, Append[ConstantArray[n, qr[[1]]], qr[[2]] /. 0 -> Nothing]], tho the outer list is not packed. $\endgroup$ – J. M. will be back soon Oct 7 '16 at 14:07
  • $\begingroup$ Yes, Internal`PartitionRagged does not bother trying to pack the outer list for partitions of the same size. I used Partition for this situation to meet OP's requirements. $\endgroup$ – user31159 Oct 7 '16 at 14:18
7
$\begingroup$

I think that the answer for Question 1 is that Partition doesn't only return packed subarrays, it returns a complete packed array.

Developer`PackedArrayQ @ Partition[arr, 25000]
(* True *)

When it can't keep everything packed, it gives up and fully unpacks everything. This of course doesn't mean that it couldn't in principle behave in a smarter way.

The best solutions I found to Question 2 are unfortunately still not very simple, so suggestions are welcome. I didn't find the solution to be trivial, so I thought it is worth sharing:

Clear[part]
part[arr_, n_] /; Divisible[Length[arr], n] := Partition[arr, n]
part[arr_, n_] :=
 Join[
  Partition[arr, n, n, {1, -1}],
  {Take[arr, -Mod[Length[arr], n]]}
 ]

This seems simple, but you may simplify this further to use Append[a, b] instead of Append[a, {el}]. It turns out that Append is another one of those unpack-eager functions, so using Join was critical. If we want Append, we must unpack to level 1 only manually:

part[arr_, n_] :=
 Append[
  Developer`FromPackedArray[Partition[arr, n, n, {1, -1}], 1],
  Take[arr, -Mod[Length[arr], n]]
 ]
$\endgroup$
  • $\begingroup$ part[Range[12], 4] seems to return a dangling empty list for me, and Developer`PackedArrayQ[part[Range[12], 4]] thus returns False. $\endgroup$ – J. M. will be back soon Oct 7 '16 at 12:26
  • $\begingroup$ @J.M. Thanks. I'm not in a good shape today. $\endgroup$ – Szabolcs Oct 7 '16 at 12:37
  • $\begingroup$ For what it's worth you can unpack one level using merely List @@. $\endgroup$ – Mr.Wizard Oct 7 '16 at 13:00
  • $\begingroup$ Your second function is still leaving dangling lists: part[{1, 2, 3}, 3] -> {{1, 2, 3}, {}} $\endgroup$ – Mr.Wizard Oct 7 '16 at 13:17
  • 1
    $\begingroup$ @Mr.Wizard It still needs the Divisible test. $\endgroup$ – Szabolcs Oct 7 '16 at 13:25
6
$\begingroup$

I hesitated to post this as it is essentially a re-post of my answer to Reversibly merging sets of $k$ adjacent elements in an array but it occurs to me that this is a better place for that answer anyway.

My work-around was similar to your own except that instead of using Join I used a different padding specification and then trimmed the trailing list with a Part assignment. Improving(?) that to use MapAt instead (in operator form) gives a fairly clean function:

fn[a_, k_Integer] :=
  List @@ Partition[a, k, k, 1] //
    MapAt[# ~Drop~ Mod[Length @ a , k, 1 - k] &, -1]

Test:

Table[fn[Range@i, 3], {i, 9}]
{
 {{1}},
 {{1, 2}},
 {{1, 2, 3}},
 {{1, 2, 3}, {4}},
 {{1, 2, 3}, {4, 5}},
 {{1, 2, 3}, {4, 5, 6}},
 {{1, 2, 3}, {4, 5, 6}, {7}},
 {{1, 2, 3}, {4, 5, 6}, {7, 8}},
 {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}}
}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.