Can LibraryLink receive and output a list of strings?

Question

I need to use libraryLink to process a list of strings, I haven't found a suitable example, confused about the types of the arguments and return types. Does libraryLink directly support it, which one is suitable for me?

Answer 1

You have three options:

1. Use MathLink for argument passing/return. Read "LinkObject Arguments and Result" in the LibraryLink documentation. Anything that can be represented as a Mathematica expression can be sent through MathLink, so this is the easiest way. The drawback is (1) performance (2) some parts of the MathLink API which would allow better performance are not documented. (This is an example of a situation where the lack of documentation caused a bit of frustration.)
2. Serialize the list of strings yourself in some way, and then deserialize in Mathematica. For example:
   • If your strings have no newlines, you can use newlines as a separator and concatenate into a single string. Then use StringSplit in Mathematica.
   • If your strings have no null characters, you can use the null character as separator and encode the string into a byte NumericArray.
   • You could just encode everything into JSON and use Developer`ReadRawJSONString to deserialize.
   • You could encode everything into WXF, transfer as a NumericArray, then deserialize with either BinaryDeserialize or Developer`ReadWXFByteArray. You will need to implement your own WFX writer. This is not infeasible, but it'll take a bit of work. See the WXF Specification. Back in 2018, a Wolfram developer mentioned that they might make a C WXF encoder library available, but I haven't heard anything since then. See WXF encoder library for C++
3. Store your list of strings on the C side, and retrieve each list element with a separate function call, using a final function call to clean up. I think this is a good approach when you want to return a small, fixed number of heterogeneous data structures. I would not do this if I had to return an indeterminate number of homogeneous pieces of data.

If you don't need top performance, my recommendation is to go with MathLink argument passing. I use this for many functions in my IGraph/M package. For a few other functions, which transfer a very large amount of data (thus the bottleneck is actually data transfer), I used a naive serialization of ragged integer matrices into a flat integer list.

See also:

• MathLink (WSTP) vs LibraryLink performance
• Returning multiple results from a LibraryLink function

Answer 2

Some time ago I wrote a post on community.wolfram.com about integrating C++ with the Wolfram Language using a library called LibraryLink Utilities (LLU). The post can be found here

https://community.wolfram.com/groups/-/m/t/2133603

I recommend reading the whole thing if you are interested in the topic. Below I will paste the section about exchanging lists of data between C/C++ and WL, which I hope answers your question directly. The text below contains references to LLU which may not be clear if you are not familiar with the library but you can simply skip those and focus on the parts that refer to the "pure" LibraryLink.

Passing and returning heterogeneous lists

LibraryLink has a fixed list of types that can be used as library function arguments and return values. For instance, see the "Details and Options" section in the documentation of LibraryFunctionLoad. As you can see, it is straightforward to pass an array of numeric data, a string or an image but already sending a list of images is hard. One can conform all the images and transfer them via LibraryLink as a single Image3D but this puts a toll on efficiency. Sending a list of strings is simply impossible without tricks or workarounds.

This topic has been brought up by the community quite a few times:

• Returning multiple results from a LibraryLink function
• Is it possible to let LibraryLink return multiple results?
• LibraryFunctionLoad for multivariate C++ function containing several arguments
• LibraryFunctionLoad with variable number of arguments
• Working with a variable number of arguments in LibraryLink

It turns out that the solution to this problem already exists in LibraryLink and is called DataStore. The only problem is that for now it remains undocumented. Below is a quick glimpse of what DataStore is, which should also help understand the C++ wrappers of DataStore that LLU has to offer.

As can be seen in the picture, DataStore is a simple unidirectional linked list but with limited functionality exposed, compared for instance to std::forward_list. The API provided in the Wolfram Library allows for

• creating an empty DataStore
• copying or deleting existing DataStore
• appending new nodes at the end
• iterating over nodes
• obtaining the length of the store (in constant time)

Each node of the DataStore carries a value of type MArgument, which is a union type of all types that LibraryLink can handle as function arguments or return values. The MArgument union include DataStore which means that the store can be nested. Additionally, every node contains an optional "name" which can be any string. Names do not have to be unique.

A simple library function that takes a list of strings (in the form of a DataStore) and returns a new list with each of the input strings reversed, implemented in pure LibraryLink may look like this:

EXTERN_C DLLEXPORT int StringsReversedLibraryLink(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {

    // get DataStore which is the first input argument to the library function
    DataStore ds_in = MArgument_getDataStore(Args[0]);

    mint length = libData->ioLibraryFunctions->DataStore_getLength(ds_in);

    // create new DataStore to hold the result
    DataStore ds_out = libData->ioLibraryFunctions->createDataStore();
    if (ds_out == nullptr) {
        // error handling
    }
    
    // start traversing the DataStore from the first node
    DataStoreNode dsn = libData->ioLibraryFunctions->DataStore_getFirstNode(ds_in);
    while (dsn != nullptr) {
        MArgument node_data;
        if (libData->ioLibraryFunctions->DataStoreNode_getData(dsn, &node_data) != 0) {
           // error handling and cleanup
        }
        if (libData->ioLibraryFunctions->DataStoreNode_getDataType(dsn) != MType_UTF8String) {
           // error handling and cleanup
        }
        // reverse the order of characters in the string and push to the output DataStore
        std::string_view s {MArgument_getUTF8String(node_data)};
        std::string outStr(s.rbegin(), s.rend());    // create reversed copy
        libData->ioLibraryFunctions->DataStore_addString(ds_out, outStr.data());

        // move to the next node
        dsn = libData->ioLibraryFunctions->DataStoreNode_getNextNode(dsn);
    }
    // set the newly created DataStore as the result of this library function call
    MArgument_setDataStore(Res, ds_out);

    return LIBRARY_NO_ERROR;
}

It is clear how to operate on DataStores in C code now, but the Wolfram Language side of LibraryLink also uses a representation of this structure and it is defined as follows:

Developer`DataStore[node_expr$1, node_expr$2, ..., node_expr$n]

where each node_expr is of the form string -> expr or just expr with an extra requirement that expr must be an expression supported in LibraryLink.

For example, Developer`DataStore["abc", "de", "f"] passed to the function implemented above would result in Developer`DataStore["cba", "ed", "f"] being returned from the library.

LLU provides two direct wrappers over DataStore: LLU::GenericDataList and LLU::DataList<T>. The former equips DataStore with a proper container interface including methods like push_back(), front(), back() or length(). It also offers easier iteration over the list with begin() and end().

The second wrapper is templated with a node type and should be used whenever we expect a homogeneous DataStore (with all nodes of the same type). The types that can be passed as template parameter are not the raw LibraryLink types included in the MArgument union but rather their LLU counterparts. For convenience, they are enclosed in LLU::NodeType namespace with following members:

/// Boolean type, corresponds to True or False in the Wolfram Language
using Boolean = bool;

/// Machine integer type
using Integer = mint;

/// Double precision floating point type
using Real = double;

/// Complex number type, bitwise-compatible with mcomplex defined in WolframLibrary.h
using Complex = std::complex<double>;

/// Tensor stands for a GenericTensor - type agnostic wrapper over MTensor
using Tensor = MContainer<MArgumentType::Tensor>;

/// SparseArray type corresponds to the "raw" MSparseArray as LLU does not have its own wrapper for this structure yet
using SparseArray = MSparseArray;

/// NumericArray stands for a GenericNumericArray - type agnostic wrapper over MNumericArray
using NumericArray = MContainer<MArgumentType::NumericArray>;

/// Image stands for a GenericImage - type agnostic wrapper over MImage
using Image = MContainer<MArgumentType::Image>;

/// String values from LibraryLink (char*) are wrapped in std::string_view
using UTF8String = std::string_view;

/// DataStore stands for a GenericDataList - type agnostic wrapper over DataStore
using DataStore = MContainer<MArgumentType::DataStore>;

Additionally, there is also LLU::NodeType::Any which can be used to make LLU::DataList work with a heterogeneous DataStore. LLU::DataList compared to LLU::GenericDataList provides more iteration options (iteration over node values or node names only) and a function to immediately create a std::vector out of the stored values.

Let us implement the same function as above using LLU::GenericDataList:

EXTERN_C DLLEXPORT int StringsReversedGeneric(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
    auto err = LLU::ErrorCode::NoError;
    try {
        using NodeT = LLU::NodeType::UTF8String;

        LLU::MArgumentManager mngr(libData, Argc, Args, Res);
        auto dsIn = mngr.getGenericDataList(0);
        GenericDataList dsOut;
        // iterate over the GeneridDataList with a range-based for loop
        for (auto node : dsIn) {
            // we are dealing with generic DataList, so we need to be explicit about the actual node type
            std::string_view s = node.as<NodeT>();
            std::string reversed {s.rbegin(), s.rend()};    // create reversed copy
            dsOut.push_back(std::string_view(reversed));    // passing a view is fine because DataStore will copy the string immediately
        }
        
        mngr.set(dsOut);
    } catch (const LLU::LibraryLinkError& e) {
        err = e.which();
    } catch (...) {
        err = LLU::ErrorCode::FunctionError;
    }
    return err;

In the function above we know that all nodes should contain strings, so we could use LLU::DataList<LLU::NodeType::UTF8String> instead of the generic data list. The code would stay almost the same, except we could simply call node.value() to get the corresponding std::string_view.

For a different example, consider a function SeparateKeysAndValue that takes a data store of named complex numbers and separates it into two data stores: one holding names of the original data store and another one with values. For instance,

Developer`DataStore["a" -> 1 + 2.5 * I, "b" -> -3. - 6.I, 2I, "d" -> -4]

would be transformed into

Developer`DataStore[
    "Keys" -> Developer`DataStore["a", "b", "", "d"], 
    "Values" -> Developer`DataStore[1. + 2.5 * I, -3. - 6.I, 2.I, -4.]
]

The implementation of such library function may look as follows:

EXTERN_C DLLEXPORT int SeparateKeysAndValues(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
    LLU::MArgumentManager mngr(libData, Argc, Args, Res);

    auto dsIn = mngr.getDataList<LLU::NodeType::Complex>(0);
    DataList<LLU::NodeType::UTF8String> keys;
    DataList<LLU::NodeType::Complex> values;

    // we use structured bindings to immediately split each node into name and value
    for (auto [name, value] : dsIn) {
        keys.push_back(name);
        values.push_back(value);
    }

    DataList<GenericDataList> dsOut; // the output type is a DataList of DataLists
    dsOut.push_back("Keys", std::move(keys));
    dsOut.push_back("Values", std::move(values));

    mngr.set(dsOut);
}

Instead of a single loop with structured bindings, we could also make two passes over the input data list:

DataList<LLU::NodeType::UTF8String> keys;
for (auto name : LLU::NameAdaptor {dsIn}) {
    keys.push_back(name);
}
    
DataList<LLU::NodeType::Complex> values;
for (auto value : LLU::ValueAdaptor {dsIn}) {
    values.push_back(value);
}

Notice that we use iterator adaptors LLU::NameAdaptor and LLU::ValueAdaptor to iterate over a specific property of the nodes (either name or value, respectively).