3
$\begingroup$

Suppose I have a binary file which consists of a list of integer or real numbers. Let's say signed 8-bit integers; so each byte of the file is a single such integer. How can I read it and turn it into a NumericArray with Integer8? (without going through a List, obviously). I could not find anything like that in the documentation. The closest I found is to read the file as ByteArray and then try to use what's suggested in the last answer of How to convert a RawArray of bytes to ByteArray? to convert to NumericArray. Unfortunately this only seems to work with Byte, i.e., unsigned 8-bit integers:

test = ReadByteArray["data/set0-config"];
test2 = NumericArray[test, "Integer8"]

fails with the unhelpful error message

NumericArray::nconvsa: The argument ByteArray[...] cannot be automatically converted to a NumericArray of type Integer8. Try using one of the following conversion methods: "Round", "ClipAndCoerce" or "ClipAndRound".

(none of these methods achieve the desired result, not to mention the fact that I don't want Mathematica to rewrite the whole binary data).

Overall, I've found very little information on NumericArray (e.g., another obvious question is whether they can be used in compiled functions).

$\endgroup$
  • $\begingroup$ Isn't this precisely what BinaryReadList does? $\endgroup$ – Roman Jun 4 at 9:34
  • $\begingroup$ @Roman BinaryReadList returns a List, i.e. a representation that is neither efficient, nor one-to-one. OP wants a function which returns a NumericArray corresponding to the type being read. $\endgroup$ – Szabolcs Jun 4 at 12:26
  • $\begingroup$ Does NumericArray[test, "UnsignedInteger8"] fix things for you? $\endgroup$ – Chip Hurst Jun 4 at 12:36
  • $\begingroup$ @Szabolcs BinaryReadList returns a packed array, not a list, and I thought that would be quite efficient. Granted, the ByteCount is about 8 bytes per number, so it's not one-to-one on a binary level; but for speed purposes it's pretty good. Otherwise, ByteArray@BinaryReadList[name, "Byte"]? $\endgroup$ – Roman Jun 4 at 13:32
  • 1
    $\begingroup$ @Roman With NumericArray being so new (and not yet very well integrated), no one is very informed right now ... I found only limited use for it as a way to hold data that is manipulated by LibraryLink functions written in C. To manipulate the data directly in Mathematica, the best option seems to be to unpack it to a normal List, except for a few trivial kinds of manipulations like taking part of the array. $\endgroup$ – Szabolcs Jun 4 at 13:52
4
$\begingroup$

You can read the binary data in the NumericArray (or PackedArray) using deserialization from the WXF format. To do this, simply add a header on the fly in front of the binary data.

WXF Format Description

Example:

(* Create Binary File *)
data1 = N@Table[Sin[x], {x, 0, 1000 Pi, Pi/1000}];
data2 = N@Table[(Sin[x] + 0.25 Sin[11 x]), {x, 0, 1000 Pi, Pi/1000}];
file = File["data.binary"];
BinaryWrite[file, Join[data1, data2], "Real32"];
Close[file];

(* Read Binary File as List  *)
RepeatedTiming[
  dataStd = 
    ArrayReshape[BinaryReadList[file, "Real32"], {2, Length@data1}];
  ][[1]]

(* Convert array length to byte list (Length Encoding (Varint)) *)
bits = IntegerDigits[2 Length@data1, 2];
grouped7 = Partition[Reverse[bits], UpTo[7]];
grouped8 = Map[Composition[PadLeft[#, 8] &, Reverse], grouped7];
varint = ReplacePart[grouped8, {i_, 1} /; i < Length[grouped8] :> 1];
bl = Map[FromDigits[#, 2] &, varint];

(* Read Binary File as PackedArray **(Code = 193)** *)
RepeatedTiming[
  ba = Join[ByteArray[{56, 58, 193, 34, 1}], ByteArray[bl], 
    ReadByteArray[file]];
  dataPA = ArrayReshape[BinaryDeserialize[ba], {2, Length@data1}];
  ][[1]]

(* Read Binary File as NumericArray **(Code = 194)** *)
    RepeatedTiming[
  ba = Join[ByteArray[{56, 58, 194, 34, 1}], ByteArray[bl], 
    ReadByteArray[file]];
  dataNA = ArrayReshape[BinaryDeserialize[ba], {2, Length@data1}];
  ][[1]]

(* Test Data Type *)
Developer`PackedArrayQ[dataPA]
NumericArrayQ[dataNA]

(* Test Data *)
ListPlot[{Take[data1, 1000], Take[data2, 1000]}]
ListPlot[{Take[dataStd[[1]], 1000], Take[dataStd[[2]], 1000]}]
ListPlot[{Take[dataPA[[1]], 1000], Take[dataPA[[2]], 1000]}]
ListPlot[{Take[dataNA[[1]], 1000], Take[dataNA[[2]], 1000]}]

Timing result:

List          0.034
PackedArray   0.011
NumericArray  0.0092
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.