# Read NumericArray from a file

Suppose I have a binary file which consists of a list of integer or real numbers. Let's say signed 8-bit integers; so each byte of the file is a single such integer. How can I read it and turn it into a NumericArray with Integer8? (without going through a List, obviously). I could not find anything like that in the documentation. The closest I found is to read the file as ByteArray and then try to use what's suggested in the last answer of How to convert a RawArray of bytes to ByteArray? to convert to NumericArray. Unfortunately this only seems to work with Byte, i.e., unsigned 8-bit integers:

test = ReadByteArray["data/set0-config"];
test2 = NumericArray[test, "Integer8"]


fails with the unhelpful error message

NumericArray::nconvsa: The argument ByteArray[...] cannot be automatically converted to a NumericArray of type Integer8. Try using one of the following conversion methods: "Round", "ClipAndCoerce" or "ClipAndRound".

(none of these methods achieve the desired result, not to mention the fact that I don't want Mathematica to rewrite the whole binary data).

Overall, I've found very little information on NumericArray (e.g., another obvious question is whether they can be used in compiled functions).

• Isn't this precisely what BinaryReadList does? – Roman Jun 4 '19 at 9:34
• @Roman BinaryReadList returns a List, i.e. a representation that is neither efficient, nor one-to-one. OP wants a function which returns a NumericArray corresponding to the type being read. – Szabolcs Jun 4 '19 at 12:26
• Does NumericArray[test, "UnsignedInteger8"] fix things for you? – Chip Hurst Jun 4 '19 at 12:36
• @Szabolcs BinaryReadList returns a packed array, not a list, and I thought that would be quite efficient. Granted, the ByteCount is about 8 bytes per number, so it's not one-to-one on a binary level; but for speed purposes it's pretty good. Otherwise, ByteArray@BinaryReadList[name, "Byte"]? – Roman Jun 4 '19 at 13:32
• @Roman With NumericArray being so new (and not yet very well integrated), no one is very informed right now ... I found only limited use for it as a way to hold data that is manipulated by LibraryLink functions written in C. To manipulate the data directly in Mathematica, the best option seems to be to unpack it to a normal List, except for a few trivial kinds of manipulations like taking part of the array. – Szabolcs Jun 4 '19 at 13:52

You can read the binary data in the NumericArray (or PackedArray) using deserialization from the WXF format. To do this, simply add a header on the fly in front of the binary data.

WXF Format Description

Example:

(* Create Binary File *)
data1 = N@Table[Sin[x], {x, 0, 1000 Pi, Pi/1000}];
data2 = N@Table[(Sin[x] + 0.25 Sin[11 x]), {x, 0, 1000 Pi, Pi/1000}];
file = File["data.binary"];
BinaryWrite[file, Join[data1, data2], "Real32"];
Close[file];

(* Read Binary File as List  *)
RepeatedTiming[
dataStd =
][[1]]

(* Convert array length to byte list (Length Encoding (Varint)) *)
bits = IntegerDigits[2 Length@data1, 2];
grouped7 = Partition[Reverse[bits], UpTo[7]];
grouped8 = Map[Composition[PadLeft[#, 8] &, Reverse], grouped7];
varint = ReplacePart[grouped8, {i_, 1} /; i < Length[grouped8] :> 1];
bl = Map[FromDigits[#, 2] &, varint];

(* Read Binary File as PackedArray **(Code = 193)** *)
RepeatedTiming[
ba = Join[ByteArray[{56, 58, 193, 34, 1}], ByteArray[bl],
dataPA = ArrayReshape[BinaryDeserialize[ba], {2, Length@data1}];
][[1]]

(* Read Binary File as NumericArray **(Code = 194)** *)
RepeatedTiming[
ba = Join[ByteArray[{56, 58, 194, 34, 1}], ByteArray[bl],
dataNA = ArrayReshape[BinaryDeserialize[ba], {2, Length@data1}];
][[1]]

(* Test Data Type *)
DeveloperPackedArrayQ[dataPA]
NumericArrayQ[dataNA]

(* Test Data *)
ListPlot[{Take[data1, 1000], Take[data2, 1000]}]
ListPlot[{Take[dataStd[[1]], 1000], Take[dataStd[[2]], 1000]}]
ListPlot[{Take[dataPA[[1]], 1000], Take[dataPA[[2]], 1000]}]
ListPlot[{Take[dataNA[[1]], 1000], Take[dataNA[[2]], 1000]}]


Timing result:

List          0.034
PackedArray   0.011
NumericArray  0.0092
`