Partition array without unpacking

What is the simplest way to partition a list into equal-length sublists, allow the last sublist to be shorter of necessary, and avoid unpacking?

Partition doesn't usually unpack:

arr = RandomReal[1, 100000];

DeveloperPackedArrayQ[arr]
(* True *)

DeveloperPackedArrayQ /@ Partition[arr, 25000]
(* {True, True, True, True} *)


But it does if we allow the last sublist to be shorter:

DeveloperPackedArrayQ /@ Partition[arr, UpTo@30000]
(* {False, False, False, False} *)

DeveloperPackedArrayQ /@ Partition[arr, 30000, 30000, {1, 1}, {}]
(* {False, False, False, False} *)


Question 1: Why isn't the result packed? Is there any reason I didn't think of or is this case just not optimized?

Question 2: What's an elegant way to avoid this problem?

• It's nice to see this old issue getting some attention. My work-around is pretty much the same as yours: mathematica.stackexchange.com/a/28428/121 Commented Oct 7, 2016 at 12:46
• Related: (58215) Commented Oct 21, 2016 at 4:25

One could use InternalPartitionRagged to get packed subarrays.

Code.

partition[l_List, n_Integer] := With[{qr = QuotientRemainder[Length[l], n]},

If[qr[[2]] == 0,
Partition[l, n],
InternalPartitionRagged[l, Append[ConstantArray[n, qr[[1]]], qr[[2]]]]
]

];


Usage.

list = Range[100];

p1 = partition[list, 25];
Length /@ p1
DeveloperPackedArrayQ /@ p1
(* {25, 25, 25, 25} *)
(* {True, True, True, True} *)

p2 = partition[list, 30];
Length /@ p2
DeveloperPackedArrayQ /@ p2
(* {30, 30, 30, 10} *)
(* {True, True, True, True} *)

• Alternatively, InternalPartitionRagged[l, Append[ConstantArray[n, qr[[1]]], qr[[2]] /. 0 -> Nothing]], tho the outer list is not packed. Commented Oct 7, 2016 at 14:07
• Yes, InternalPartitionRagged does not bother trying to pack the outer list for partitions of the same size. I used Partition for this situation to meet OP's requirements.
– user31159
Commented Oct 7, 2016 at 14:18

I think that the answer for Question 1 is that Partition doesn't only return packed subarrays, it returns a complete packed array.

DeveloperPackedArrayQ @ Partition[arr, 25000]
(* True *)


When it can't keep everything packed, it gives up and fully unpacks everything. This of course doesn't mean that it couldn't in principle behave in a smarter way.

The best solutions I found to Question 2 are unfortunately still not very simple, so suggestions are welcome. I didn't find the solution to be trivial, so I thought it is worth sharing:

Clear[part]
part[arr_, n_] /; Divisible[Length[arr], n] := Partition[arr, n]
part[arr_, n_] :=
Join[
Partition[arr, n, n, {1, -1}],
{Take[arr, -Mod[Length[arr], n]]}
]


This seems simple, but you may simplify this further to use Append[a, b] instead of Append[a, {el}]. It turns out that Append is another one of those unpack-eager functions, so using Join was critical. If we want Append, we must unpack to level 1 only manually:

part[arr_, n_] :=
Append[
DeveloperFromPackedArray[Partition[arr, n, n, {1, -1}], 1],
Take[arr, -Mod[Length[arr], n]]
]

• part[Range[12], 4] seems to return a dangling empty list for me, and DeveloperPackedArrayQ[part[Range[12], 4]] thus returns False. Commented Oct 7, 2016 at 12:26
• @J.M. Thanks. I'm not in a good shape today. Commented Oct 7, 2016 at 12:37
• For what it's worth you can unpack one level using merely List @@. Commented Oct 7, 2016 at 13:00
• Your second function is still leaving dangling lists: part[{1, 2, 3}, 3] -> {{1, 2, 3}, {}} Commented Oct 7, 2016 at 13:17
• @Mr.Wizard It still needs the Divisible test. Commented Oct 7, 2016 at 13:25

I hesitated to post this as it is essentially a re-post of my answer to Reversibly merging sets of $k$ adjacent elements in an array but it occurs to me that this is a better place for that answer anyway.

My work-around was similar to your own except that instead of using Join I used a different padding specification and then trimmed the trailing list with a Part assignment. Improving(?) that to use MapAt instead (in operator form) gives a fairly clean function:

fn[a_, k_Integer] :=
List @@ Partition[a, k, k, 1] //
MapAt[# ~Drop~ Mod[Length @ a , k, 1 - k] &, -1]


Test:

Table[fn[Range@i, 3], {i, 9}]

{
{{1}},
{{1, 2}},
{{1, 2, 3}},
{{1, 2, 3}, {4}},
{{1, 2, 3}, {4, 5}},
{{1, 2, 3}, {4, 5, 6}},
{{1, 2, 3}, {4, 5, 6}, {7}},
{{1, 2, 3}, {4, 5, 6}, {7, 8}},
{{1, 2, 3}, {4, 5, 6}, {7, 8, 9}}
}
`