WXF is a fully documented binary encoding for Wolfram Language expressions. WRI has already published a Python encoder/decoder for this format. ExternalEvaluate also uses it.

Has anyone developed a WXF encoder for C++?

Unlike in Python, in C++ there's no standard set of data structures to use to represent a WL expression. Therefore I expect that eventually there will be several C++ WXF libraries, backed by different data structures and intended for different uses.

The specific use case I have in mind here is returning complex results from LibraryFunctions that cannot be simply represented as an MTensor. The typical solution for this is using MathLink. I suspect that encoding those results to WXF, transferring them as a ByteArray, then decoding with BinaryDeserialize will be faster, in some cases much faster, than using MathLink. This suspicion is based on noticing that sometimes, even encoding to JSON and transferring as a string can be faster than using MathLink.

  • 7
    $\begingroup$ The original internal WXF library was written in C. There were plans to open-source it. $\endgroup$ Oct 21 '18 at 10:24
  • 1
    $\begingroup$ One cool thing this would provide is the means to use JLink (or more relevant to me, PJLink) with WXF instead of MathLink as it’s then just basically a matter of handling the type-decoding and encoding intelligently. Nothing else about JLink really needs MathLink. This could be then be reasonably easily extended to languages like Julia, too. $\endgroup$
    – b3m2a1
    Oct 22 '18 at 16:30

We have developed a C library to support BinarySerialize/BinaryDeserialize. I confirm that we plan to open-source it as well.

The library is built around a struct, called WXFExpr, that can represent any WXF types. It decodes a stream of WXF bytes into a stream of WXFExpr, which can be seen as an intermediary representation, ensuring the encoding is consistent with WXF specifications.

It's then up to the caller of the library to turn these WXFExpr into something meaningful. That's exactly what the Wolfram Kernel does for BinaryDeserialize.

Similarly during serialization the library turns WXFExpr into a consistent stream of WXF bytes. The caller fills the WXFExpr with relevant data.

  • $\begingroup$ How does the BinarySerialize/WXF pipeline compare to MathLink performance wise in terms of transmitting huge packed arrays? $\endgroup$
    – b3m2a1
    Oct 22 '18 at 16:40
  • $\begingroup$ @b3m2a1 I believe MathLink does very well with huge packed arrays. It's almost as fast as copying the array as many times as necessary for the transfer. I expect a WXF-based transfer to do better than MathLink with complex expressions or non-packed arrays. There's also the extra benefit of transparency: since the format is documented, it is very clear what can be done with it. See all the spelunking we did a month ago just to determine if the expression on the MathLink link (1) is a packed array (2) its type. With WXF no spelunking is needed. $\endgroup$
    – Szabolcs
    Oct 22 '18 at 16:49
  • $\begingroup$ @Szabolcs true, but depending on how much better MathLink is it might still be worth having some well-designed MathLink stuff lying around. The decoding issue is not great, but can be handled pretty cleanly on the Mathematica side by providing some extra data (e.g. wrapping the packed array data with some fake head like PackedArray). This is basically the idea JLink uses. Some kind of intelligent type decoder (and maybe encoder, to make things like Graph transmit cleanly) will be necessary in the end for WXF formats anyway. $\endgroup$
    – b3m2a1
    Oct 22 '18 at 16:54
  • $\begingroup$ On the other hand, if the performance hit is relatively small, then I can imagine just ditching MathLink whenever possible. In any case, the benchmark would be nice to have. $\endgroup$
    – b3m2a1
    Oct 22 '18 at 16:55
  • 3
    $\begingroup$ @Dorian Is there any updates regarding completion of this C library? $\endgroup$
    – fmvin
    Jun 6 '19 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.