What are the main differences between RawArray and PackedArray?

Question

What I know:

• PackedArray is documented (to some extent) in "Developer Utilities Package", RawArray is just mentioned in Raster3D and ImageApply,

• RawArray supports Byte, SignedInteger8/16/32, Bit16, Real32, Real and Complex128 as entries, while PackedArray only Integer, Real and Complex(128),

• PackedArray displays as a list but RawArray as (somewhat of) a skeleton.

What I do not understand is why there are two of them. Why not just stick with the one that supports more entry types and display it as skeleton if and only if it is too long?

Also, how does ByteArray fit into the picture?

Edit: You can create a PackedArray using:

ap = Developer`ToPackedArray[{1, 2, 3}]
(* {1,2,3} *)
Developer`PackedArrayQ[ap]
(* True *)
Developer`PackedArrayForm[ap]
(* PackedArray[Integer, <3>] *)

(Note that there is in fact no PackedArray command, the head of the last output expression is a string.)

You can create a RawArray using:

ar = Developer`AllocateRawArray["Byte", {2, 2}]
(* RawArray[Byte, <2,2>] *)
Developer`RawArrayQ[ar]
(* True *)
Developer`RawArrayType[ar]
(* Byte *)

(ditto about RawArray).

You can create a ByteArray directly:

ab = ByteArray[{1,2,3}]
(* ByteArray[<3>] *)
ByteArrayQ[ab]
(* True *)

All the cross-tests (e.g., RawArrayQ done on ab) return False.

Answer 1

Here goes my conjecture but I would welcome any opinion or clarification by people in the know.

1. PackedArray stores data in a computation-efficient way. It still represents a List in all other respects, and is displayed as one. Some Lists are generated as PackedArrays silently where Mathematica decides it possible and helpful, and unpacked if an operation incompatible with the packing is performed (like changing one element for another of a different type, out of the range of the type, or for a symbolic expression). The reason to store Integers, Reals and Complexes only is that these are natively supported for computation. The actual size will also depend on the system implementation (32- or 64-bit). You can perform mathematical operations on a PackedArray and it will stay packed and will presumably vectorise the operations where possible.

2. RawArray stores data in a memory-efficient way in a wide variety of formats that may be useful for different purposes. (Storing a Byte instead of a small Integer indeed saves a lot of memory (88% on a 64-bit system) but one would need additional CPU instructions in order to calculate with bytes.) It does not represent a list as it is considered a zero-depth object: ar[[1]] will give an error (and then, unwillingly, tell you it's actually "Byte" in the OP example). ar[[2]] gives yet another special object called an ArrayObject which indeed has no structure as far as Mathematica is concerned. It follows from this that one can't extract data from the RawArray using Part, and can not perform mathematical operation with these. RawArray is ideal for (and is used for) storing special-purpose data in objects representing big amounts of very regularly structured information like bitmap images and sound samples. Only functions actually designed to deal with the specific kind of data access and modify the information stored inside, otherwise the block of data it represents is constant.

3. ByteArray is roughly half the way between these: it stores data in a memory-efficient way, with no direct computation, but still representing a list of data which can be accessed directly using Part. As opposed to the other two it is limited to a linear (one-dimensional) lists and thus perfect for compressed streams and the like. It can also be converted to a normal list using Normal, like other optimized data structures. It turns out that a ByteArray is actually identical to an ArrayObject[Byte, _], but this is not manifested externally. Also, the latter supports a wider selection of types than just Bytes.