Read NumericArray from a file

Question

Suppose I have a binary file which consists of a list of integer or real numbers. Let's say signed 8-bit integers; so each byte of the file is a single such integer. How can I read it and turn it into a NumericArray with Integer8? (without going through a List, obviously). I could not find anything like that in the documentation. The closest I found is to read the file as ByteArray and then try to use what's suggested in the last answer of How to convert a RawArray of bytes to ByteArray? to convert to NumericArray. Unfortunately this only seems to work with Byte, i.e., unsigned 8-bit integers:

test = ReadByteArray["data/set0-config"];
test2 = NumericArray[test, "Integer8"]

fails with the unhelpful error message

NumericArray::nconvsa: The argument ByteArray[...] cannot be automatically converted to a NumericArray of type Integer8. Try using one of the following conversion methods: "Round", "ClipAndCoerce" or "ClipAndRound".

(none of these methods achieve the desired result, not to mention the fact that I don't want Mathematica to rewrite the whole binary data).

Overall, I've found very little information on NumericArray (e.g., another obvious question is whether they can be used in compiled functions).

Answer 1

You can read the binary data in the NumericArray (or PackedArray) using deserialization from the WXF format. To do this, simply add a header on the fly in front of the binary data.

WXF Format Description

Example:

(* Create Binary File *)
data1 = N@Table[Sin[x], {x, 0, 1000 Pi, Pi/1000}];
data2 = N@Table[(Sin[x] + 0.25 Sin[11 x]), {x, 0, 1000 Pi, Pi/1000}];
file = File["data.binary"];
BinaryWrite[file, Join[data1, data2], "Real32"];
Close[file];

(* Read Binary File as List  *)
RepeatedTiming[
  dataStd = 
    ArrayReshape[BinaryReadList[file, "Real32"], {2, Length@data1}];
  ][[1]]

(* Convert array length to byte list (Length Encoding (Varint)) *)
bits = IntegerDigits[2 Length@data1, 2];
grouped7 = Partition[Reverse[bits], UpTo[7]];
grouped8 = Map[Composition[PadLeft[#, 8] &, Reverse], grouped7];
varint = ReplacePart[grouped8, {i_, 1} /; i < Length[grouped8] :> 1];
bl = Map[FromDigits[#, 2] &, varint];

(* Read Binary File as PackedArray **(Code = 193)** *)
RepeatedTiming[
  ba = Join[ByteArray[{56, 58, 193, 34, 1}], ByteArray[bl], 
    ReadByteArray[file]];
  dataPA = ArrayReshape[BinaryDeserialize[ba], {2, Length@data1}];
  ][[1]]

(* Read Binary File as NumericArray **(Code = 194)** *)
    RepeatedTiming[
  ba = Join[ByteArray[{56, 58, 194, 34, 1}], ByteArray[bl], 
    ReadByteArray[file]];
  dataNA = ArrayReshape[BinaryDeserialize[ba], {2, Length@data1}];
  ][[1]]

(* Test Data Type *)
Developer`PackedArrayQ[dataPA]
NumericArrayQ[dataNA]

(* Test Data *)
ListPlot[{Take[data1, 1000], Take[data2, 1000]}]
ListPlot[{Take[dataStd[[1]], 1000], Take[dataStd[[2]], 1000]}]
ListPlot[{Take[dataPA[[1]], 1000], Take[dataPA[[2]], 1000]}]
ListPlot[{Take[dataNA[[1]], 1000], Take[dataNA[[2]], 1000]}]

Timing result:

List          0.034
PackedArray   0.011
NumericArray  0.0092