# What's purpose of the new function BinarySerialize?

11.1 introduced a new function BinarySerialize, but I don't know what it can do better than the traditional method.Its behavior is very similar to Compress,though I cannot find any advantage of it.It even consumes more space than Compress,such as

BinarySerialize[Range[100]] // Normal // Length

805

Compress[Range[100]] // ToCharacterCode // Length


290

And ByteCount[BinarySerialize[Range[100]]] is also greater than ByteCount[Compress[Range[100]]].So what's purpose of this function? Anyone can provide a good example to use it?

• see how it compares if you do PerformanceGoal->"Size" – george2079 Apr 5 '17 at 12:09
• @george2079 So it is for save space?Why not set PerformanceGoal->"Size" as a default option? – yode Apr 5 '17 at 13:28

Disclaimer: This answer is written from a user's point of view. For useful insider information on this topic see this discussion with Mathematica developers on Community Forums.

Introduction

Binary serialization is rewriting expressions as an array of bytes (list of integers from Range[0, 255]). Binary representation of expression takes less space than a textual one and also can be exported and imported faster than text.

How do Compress and BinarySerialize functions work?

Compress (with default options) always does three steps:

1. It performs binary serialization.
2. It deflates result using zlib.
3. It transforms deflated result to a Base64-encoded text string.

BinarySerialize performs only binary serialization and sometimes deflates the result using zlib. With default options it will decide itself if it wants to deflate or not. With an option PerformanceGoal -> "Speed" it will avoid deflation. With an option PerformanceGoal -> "Size" it will likely deflate. BinarySerialize returns a ByteArray object. ByteArray is something like a packed array of 8-bit integers. However FullForm of ByteArray is visualized as Base64-encoded text string. This visualization can be somewhat misleading, because internally ByteArrays are stored and operated in binary form, not as text strings.

Binary serialization algorithms of Compress and BinarySerialize

Original serialization algorithm of Compress is described in this answer. That algorithm is not very optimized for size and produces larger-then-necessary output for many typical expressions. For example, it has no support for packed arrays of integers and rewrites such arrays as nested lists, which take a lot bytes.

BinarySerialize uses a more size-optimized binary serialization algorithm compared to what Compress (with default options) does. This algorithm supports packed arrays of integers, has optimizations for integers of different size (8,16,32 bit), stores big integers in binary form (not as text strings), and has other optimizations.

Applications of BinarySerialize

Using BinarySerialize we can write our own Compress-like functions with better compression. For example we can write myCompress function which does the same three steps as original Compress, but uses BinarySerialize for the serialization step:

myCompress[expr_]:=Module[
{compressedBinaryData},
compressedBinaryData = BinarySerialize[expr, PerformanceGoal->"Size"];
DeveloperEncodeBase64[compressedBinaryData]
];

myUncompress[string_]:=Module[
{binaryData},
binaryData = DeveloperDecodeBase64ToByteArray[string];
BinaryDeserialize[binaryData]
];


Even for simple integer list we can see size reduction.

Compress[Range[100]] // StringLength
(* 290 *)

myCompress[Range[100]] // StringLength
(* 244 *)

myUncompress[myCompress[Range[100]]] === Range[100]
(* True *)


If we take an expression with large number of small integers we get much more noticeable improvement:

bitmap = Rasterize[Plot[x, {x, 0, 1}]];

StringLength[Compress[bitmap]]
(*31246*)

StringLength[myCompress[bitmap]]
(*17820*)

myUncompress[myCompress[bitmap]] === bitmap
(* True *)


Conclusion

The example above shows that the result of a simple user-defined function myCompress based on a BinarySerialize can be almost twice more compact than the result of Compress.

Outlook

To decrease the output size even further one can use a compression algorithm with higher compression settings (in the second step) or use Ascii85-encoding instead of Base64 in the third step.

Appendix 1: Undocumented options of Compress

I have noticed that in Version 11.1 Compress has more undocumented options than in previous versions. Those options allows one to:

• Disable both compression and Base64 encoding and return a binary serialized result as a string with unprintable characters:

Compress[Range[100], Method -> {"Version" -> 4}]

• Change binary serialization algorithm to a more efficient one, but not exactly to BinarySerialize.

Compress[Range[100], Method -> {"Version" -> 6}] // StringLength

(* 254 *)

There is also a "ByteArray" option shown in usage message ??Compress but it does not work in Version 11.1.

Note that this behavior is undocumented and may change in future versions.

Appendix 2: Compression option of BinarySerialize

Just for fun one can manually compress result of BinarySerialize[..., PerformanceGoal -> "Speed"] to get the same output as BinarySerialize[..., PerformanceGoal -> "Size"] produces. This can be done with the following code:

myBinarySerializeSize[expr_]:=Module[
{binaryData, dataBytes, compressedBytes},
binaryData = Normal[BinarySerialize[expr, PerformanceGoal->"Speed"]];
dataBytes = Drop[binaryData, 2]; (*remove magic "7:"*)
compressedBytes = DeveloperRawCompress[dataBytes];
ByteArray[Join[ToCharacterCode["7C:"], compressedBytes]]
]


We can check that it gives the same result as PerformanceGoal -> "Size" option

data = Range[100];
myBinarySerializeSize[data] === BinarySerialize[data, PerformanceGoal -> "Size"]


Appendix 3: zlib compression functions

Description of undocumented zlib compression/decompression functions DeveloperRawCompress and DeveloperRawUncompress can be found in this answer.

Appendix 4: Base64 encoding functions

Usage of Base64 encoding/decoding functions from the Developer context can be explained using the following code:

binaryData = Range[0, 255];

Normal[
DeveloperDecodeBase64ToByteArray[
DeveloperEncodeBase64[binaryData]
]
] == binaryData

(* True *)

• You seem to have dug deep here. Perhaps you have some of the answers for this too. – Szabolcs Apr 5 '17 at 20:11
• I just feel very pity that DeveloperRawCompress just can compress string or list. – yode Apr 9 '17 at 14:34

Measure the difference in bytes with ByteCount.

BinarySerialize[Range[100]] // ByteCount

901

Range[100] // ByteCount

936


BinarySerialize will occupy fewer bytes. Not a big difference in this trivial example but the difference generally widens as the object to be serialised gets larger.

Also you must Compress both for apples-to-apples comparison.

Compress@BinarySerialize[Range[100]] // ByteCount

336

Compress@Range[100] // ByteCount

368


Hope this helps.

• So as your understand,this function is intend to save space? – yode Apr 5 '17 at 17:43
• @yode I think that's probably right. Also, and here I'm just speculating, it might be faster to revert using BinaryDeserialize. Of course the fact that it's a binary format also probably means it's not platform independent. – b3m2a1 Apr 5 '17 at 17:46
• @MB1965 You mean if I share my result of BinarySerialize to you,you cannot decode it by BinaryDeserialize? – yode Apr 5 '17 at 17:53
• @MB1965 I would expect them to state it explicitly if it is not platform independent (as it is stated for the MX format). Compress is platform independent. – Szabolcs Apr 5 '17 at 18:02
• @Szabolcs this is true... hmm... I have no Windows machines lying about but hopefully someone can test this and prove me wrong. – b3m2a1 Apr 5 '17 at 18:04

Here's an interesting complement to Edmund's answer. It's not always the case that the ByteArray form will be more compact. DocFind is just a table of reflinks I use for doc searching.

In[155]:= bs = BinarySerialize@DocFind[]; // RepeatedTiming

Out[155]= {1.8, Null}

In[156]:= cp = Compress@DocFind[]; // RepeatedTiming

Out[156]= {1.74, Null}

In[157]:= bs // ByteCount

Out[157]= 3027593

In[158]:= cp // ByteCount

Out[158]= 233104


On the other hand it's minimally faster to BinaryDeserialize:

In[159]:= Uncompress@cp; // RepeatedTiming

Out[159]= {0.34, Null}

In[160]:= BinaryDeserialize@bs; // RepeatedTiming

Out[160]= {0.30, Null}


If we use a much larger input we get a much more interesting result, though:

In[145]:= bs2 = BinarySerialize@Range[10000000]; // AbsoluteTiming

Out[145]= {0.16928, Null}

In[153]:= cp2 = Compress@Range[10000000]; // AbsoluteTiming

Out[153]= {4.92081, Null}


My intuition would suggest that BinarySerialize should almost always be as fast or faster than Compress but I'd be interested to be proven wrong.

Here the serialized version is much more compact:

In[161]:= bs2 // ByteCount

Out[161]= 80000104

In[162]:= bs3 // ByteCount

Out[162]= 31276128


And it's so much faster to use BinaryDeserialize here than Uncompress:

In[163]:= BinaryDeserialize@bs2; // RepeatedTiming

Out[163]= {0.13, Null}

In[164]:= Uncompress@cp2; // AbsoluteTiming

Out[164]= {29.8372, Null}


Again, because it's a byte format, my intuition would suggest Binary* will be faster and often more compact than a compressed string.

### Update: yode confirmed that it is platform independent

On the other hand, since it's a byte format I would also believe it's platform dependent -- i.e., I can't take my serialized ByteArray and mail it to you and expect it to work if we have different operating systems/architecture, the same way I can't expect that of .mx files.

• I'm in window 10.Can we test whether it is platform dependent or not? – yode Apr 5 '17 at 18:18
• Run this code please.NotebookPut[Uncompress[FromCharacterCode[Flatten[ImageData[Import["http://i.stack.imgur.com/A2bMu.png"],"Byte"]]]]].Do you get Range[100] by BinaryDeserialize? – yode Apr 5 '17 at 18:21
• @yode for some reason that gave me an "file not found" message. Alternatively you can pull the byte array file here (just go there and the file will download) and see if you can get that to BinaryDeserialize`. – b3m2a1 Apr 5 '17 at 18:24
• I get it.see this.It is your file? – yode Apr 5 '17 at 18:30
• @yode yep. It's platform independent. I'll amend my post. – b3m2a1 Apr 5 '17 at 18:31

In general, serialization is widely used for two purposes:

1. To send large objects "over the wire" by TCP/IP from one computer to another. For example, a company might have a programming object such as a BillOfMaterials containing a large number of part numbers. The company would serialize the BillOfMaterials, send it to a vendor, and request a quote and an estimate of parts availability. The vendor would de-serialize the BillOfMaterials, assign prices to the part numbers and indicate whether in stock, serialize the completed BillOfMaterials, and return the quote to the originating company.

2. Memoization. For recursive algorithms using bimomial (Binomial[...]) or multinomial (Multinomial[...]) coefficients, the integers involved become very large very fast. Even computers with large amounts of RAM cannot keep intermediate results in core memory and still do useful work. It is common to keep a dictionary or map (in Mathematica, an Association) of the results of computing a binomial or multinomial coefficient as a serialized object on disk. The dictionary key is a concatenation of the binomial or multinomial arguments, and the dictionary value is the file name of the serialized result. So, instead of recomputing the coefficient, it is read from disk and de-serialized. The same basic idea could be used in Dynamic Programming, in which a large problem is split into many smaller problems. Since the smaller problems may have to be solved over and over again, their solutions can be serialized to disk and recalled as needed.