15
$\begingroup$

WXF is a fully documented binary encoding for Wolfram Language expressions. WRI has already published a Python encoder/decoder for this format. ExternalEvaluate also uses it.

Has anyone developed a WXF encoder for C++?

Unlike in Python, in C++ there's no standard set of data structures to use to represent a WL expression. Therefore I expect that eventually there will be several C++ WXF libraries, backed by different data structures and intended for different uses.

The specific use case I have in mind here is returning complex results from LibraryFunctions that cannot be simply represented as an MTensor. The typical solution for this is using MathLink. I suspect that encoding those results to WXF, transferring them as a ByteArray, then decoding with BinaryDeserialize will be faster, in some cases much faster, than using MathLink. This suspicion is based on noticing that sometimes, even encoding to JSON and transferring as a string can be faster than using MathLink.

Improve this question
$\endgroup$
2
  • 7
    $\begingroup$ The original internal WXF library was written in C. There were plans to open-source it. $\endgroup$
    – Leonid Shifrin
    Commented Oct 21, 2018 at 10:24
  • 1
    $\begingroup$ One cool thing this would provide is the means to use JLink (or more relevant to me, PJLink) with WXF instead of MathLink as it’s then just basically a matter of handling the type-decoding and encoding intelligently. Nothing else about JLink really needs MathLink. This could be then be reasonably easily extended to languages like Julia, too. $\endgroup$
    – b3m2a1
    Commented Oct 22, 2018 at 16:30

1 Answer 1

Reset to default
14
$\begingroup$

We have developed a C library to support BinarySerialize/BinaryDeserialize. I confirm that we plan to open-source it as well.

The library is built around a struct, called WXFExpr, that can represent any WXF types. It decodes a stream of WXF bytes into a stream of WXFExpr, which can be seen as an intermediary representation, ensuring the encoding is consistent with WXF specifications.

It's then up to the caller of the library to turn these WXFExpr into something meaningful. That's exactly what the Wolfram Kernel does for BinaryDeserialize.

Similarly during serialization the library turns WXFExpr into a consistent stream of WXF bytes. The caller fills the WXFExpr with relevant data.

Improve this answer
$\endgroup$
8
  • $\begingroup$ How does the BinarySerialize/WXF pipeline compare to MathLink performance wise in terms of transmitting huge packed arrays? $\endgroup$
    – b3m2a1
    Commented Oct 22, 2018 at 16:40
  • $\begingroup$ @b3m2a1 I believe MathLink does very well with huge packed arrays. It's almost as fast as copying the array as many times as necessary for the transfer. I expect a WXF-based transfer to do better than MathLink with complex expressions or non-packed arrays. There's also the extra benefit of transparency: since the format is documented, it is very clear what can be done with it. See all the spelunking we did a month ago just to determine if the expression on the MathLink link (1) is a packed array (2) its type. With WXF no spelunking is needed. $\endgroup$
    – Szabolcs
    Commented Oct 22, 2018 at 16:49
  • 3
    $\begingroup$ @Dorian Is there any updates regarding completion of this C library? $\endgroup$
    – fmvin
    Commented Jun 6, 2019 at 8:57
  • 1
    $\begingroup$ Is there any update on this? $\endgroup$
    – Szabolcs
    Commented Mar 29, 2021 at 6:35
  • 1
    $\begingroup$ Another ping on this. Is it still a good idea to build on WXF for the future? My plan was to replace all use of MathLink with transferring WXF-encoded bytes. Does this sound like a good idea, i.e. is WXF going to be well supported in the future? $\endgroup$
    – Szabolcs
    Commented Jun 15, 2022 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.