I need to use libraryLink to process a list of strings, I haven't found a suitable example, confused about the types of the arguments and return types. Does libraryLink directly support it, which one is suitable for me?
2 Answers
You have three options:
- Use MathLink for argument passing/return. Read "LinkObject Arguments and Result" in the LibraryLink documentation. Anything that can be represented as a Mathematica expression can be sent through MathLink, so this is the easiest way. The drawback is (1) performance (2) some parts of the MathLink API which would allow better performance are not documented. (This is an example of a situation where the lack of documentation caused a bit of frustration.)
- Serialize the list of strings yourself in some way, and then deserialize in Mathematica. For example:
- If your strings have no newlines, you can use newlines as a separator and concatenate into a single string. Then use
StringSplit
in Mathematica. - If your strings have no null characters, you can use the null character as separator and encode the string into a byte NumericArray.
- You could just encode everything into JSON and use
Developer`ReadRawJSONString
to deserialize. - You could encode everything into WXF, transfer as a NumericArray, then deserialize with either
BinaryDeserialize
orDeveloper`ReadWXFByteArray
. You will need to implement your own WFX writer. This is not infeasible, but it'll take a bit of work. See the WXF Specification. Back in 2018, a Wolfram developer mentioned that they might make a C WXF encoder library available, but I haven't heard anything since then. See WXF encoder library for C++
- If your strings have no newlines, you can use newlines as a separator and concatenate into a single string. Then use
- Store your list of strings on the C side, and retrieve each list element with a separate function call, using a final function call to clean up. I think this is a good approach when you want to return a small, fixed number of heterogeneous data structures. I would not do this if I had to return an indeterminate number of homogeneous pieces of data.
If you don't need top performance, my recommendation is to go with MathLink argument passing. I use this for many functions in my IGraph/M package. For a few other functions, which transfer a very large amount of data (thus the bottleneck is actually data transfer), I used a naive serialization of ragged integer matrices into a flat integer list.
See also:
Some time ago I wrote a post on community.wolfram.com about integrating C++ with the Wolfram Language using a library called LibraryLink Utilities (LLU). The post can be found here
https://community.wolfram.com/groups/-/m/t/2133603
I recommend reading the whole thing if you are interested in the topic. Below I will paste the section about exchanging lists of data between C/C++ and WL, which I hope answers your question directly. The text below contains references to LLU which may not be clear if you are not familiar with the library but you can simply skip those and focus on the parts that refer to the "pure" LibraryLink.
Passing and returning heterogeneous lists
LibraryLink has a fixed list of types that can be used as library function arguments and return values. For instance, see the "Details and Options" section in
the documentation of LibraryFunctionLoad. As you can see, it is straightforward to
pass an array of numeric data, a string or an image but already sending a list of images is hard. One can conform all the images and transfer them via
LibraryLink as a single Image3D
but this puts a toll on efficiency. Sending a list of strings is simply impossible without tricks or workarounds.
This topic has been brought up by the community quite a few times:
- Returning multiple results from a LibraryLink function
- Is it possible to let LibraryLink return multiple results?
- LibraryFunctionLoad for multivariate C++ function containing several arguments
- LibraryFunctionLoad with variable number of arguments
- Working with a variable number of arguments in LibraryLink
It turns out that the solution to this problem already exists in LibraryLink and is called DataStore
. The only problem is that for now it remains
undocumented. Below is a quick glimpse of what DataStore
is, which should also help understand the C++ wrappers of DataStore
that LLU has to offer.
As can be seen in the picture, DataStore
is a simple unidirectional linked list but with limited functionality exposed, compared for instance to
std::forward_list
. The API provided in the Wolfram Library allows for
- creating an empty
DataStore
- copying or deleting existing
DataStore
- appending new nodes at the end
- iterating over nodes
- obtaining the length of the store (in constant time)
Each node of the DataStore
carries a value of type MArgument
, which is a union type of all types that LibraryLink can handle as function arguments or
return values. The MArgument
union include DataStore
which means that the store can be nested. Additionally, every node contains an optional "name" which
can be any string. Names do not have to be unique.
A simple library function that takes a list of strings (in the form of a DataStore
) and returns a new list with each of the input strings reversed,
implemented in pure LibraryLink may look like this:
EXTERN_C DLLEXPORT int StringsReversedLibraryLink(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
// get DataStore which is the first input argument to the library function
DataStore ds_in = MArgument_getDataStore(Args[0]);
mint length = libData->ioLibraryFunctions->DataStore_getLength(ds_in);
// create new DataStore to hold the result
DataStore ds_out = libData->ioLibraryFunctions->createDataStore();
if (ds_out == nullptr) {
// error handling
}
// start traversing the DataStore from the first node
DataStoreNode dsn = libData->ioLibraryFunctions->DataStore_getFirstNode(ds_in);
while (dsn != nullptr) {
MArgument node_data;
if (libData->ioLibraryFunctions->DataStoreNode_getData(dsn, &node_data) != 0) {
// error handling and cleanup
}
if (libData->ioLibraryFunctions->DataStoreNode_getDataType(dsn) != MType_UTF8String) {
// error handling and cleanup
}
// reverse the order of characters in the string and push to the output DataStore
std::string_view s {MArgument_getUTF8String(node_data)};
std::string outStr(s.rbegin(), s.rend()); // create reversed copy
libData->ioLibraryFunctions->DataStore_addString(ds_out, outStr.data());
// move to the next node
dsn = libData->ioLibraryFunctions->DataStoreNode_getNextNode(dsn);
}
// set the newly created DataStore as the result of this library function call
MArgument_setDataStore(Res, ds_out);
return LIBRARY_NO_ERROR;
}
It is clear how to operate on DataStore
s in C code now, but the Wolfram Language side of LibraryLink also uses a representation of this structure and it is
defined as follows:
Developer`DataStore[node_expr$1, node_expr$2, ..., node_expr$n]
where each node_expr
is of the form string -> expr
or just expr
with an extra requirement that expr
must be an expression supported in LibraryLink.
For example, Developer`DataStore["abc", "de", "f"]
passed to the function implemented above would result in Developer`DataStore["cba", "ed", "f"]
being
returned from the library.
LLU provides two direct wrappers over DataStore
: LLU::GenericDataList
and LLU::DataList<T>
. The former equips DataStore
with a proper container
interface including methods like push_back()
, front()
, back()
or length()
. It also offers easier iteration over the list with begin()
and end()
.
The second wrapper is templated with a node type and should be used whenever we expect a homogeneous DataStore
(with all nodes of the same type). The types
that can be passed as template parameter are not the raw LibraryLink types included in the MArgument union but rather their LLU counterparts. For convenience,
they are enclosed in LLU::NodeType
namespace with following members:
/// Boolean type, corresponds to True or False in the Wolfram Language
using Boolean = bool;
/// Machine integer type
using Integer = mint;
/// Double precision floating point type
using Real = double;
/// Complex number type, bitwise-compatible with mcomplex defined in WolframLibrary.h
using Complex = std::complex<double>;
/// Tensor stands for a GenericTensor - type agnostic wrapper over MTensor
using Tensor = MContainer<MArgumentType::Tensor>;
/// SparseArray type corresponds to the "raw" MSparseArray as LLU does not have its own wrapper for this structure yet
using SparseArray = MSparseArray;
/// NumericArray stands for a GenericNumericArray - type agnostic wrapper over MNumericArray
using NumericArray = MContainer<MArgumentType::NumericArray>;
/// Image stands for a GenericImage - type agnostic wrapper over MImage
using Image = MContainer<MArgumentType::Image>;
/// String values from LibraryLink (char*) are wrapped in std::string_view
using UTF8String = std::string_view;
/// DataStore stands for a GenericDataList - type agnostic wrapper over DataStore
using DataStore = MContainer<MArgumentType::DataStore>;
Additionally, there is also LLU::NodeType::Any
which can be used to make LLU::DataList
work with a heterogeneous DataStore
. LLU::DataList
compared to
LLU::GenericDataList
provides more iteration options (iteration over node values or node names only) and a function to immediately create a std::vector
out
of the stored values.
Let us implement the same function as above using LLU::GenericDataList
:
EXTERN_C DLLEXPORT int StringsReversedGeneric(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
auto err = LLU::ErrorCode::NoError;
try {
using NodeT = LLU::NodeType::UTF8String;
LLU::MArgumentManager mngr(libData, Argc, Args, Res);
auto dsIn = mngr.getGenericDataList(0);
GenericDataList dsOut;
// iterate over the GeneridDataList with a range-based for loop
for (auto node : dsIn) {
// we are dealing with generic DataList, so we need to be explicit about the actual node type
std::string_view s = node.as<NodeT>();
std::string reversed {s.rbegin(), s.rend()}; // create reversed copy
dsOut.push_back(std::string_view(reversed)); // passing a view is fine because DataStore will copy the string immediately
}
mngr.set(dsOut);
} catch (const LLU::LibraryLinkError& e) {
err = e.which();
} catch (...) {
err = LLU::ErrorCode::FunctionError;
}
return err;
In the function above we know that all nodes should contain strings, so we could use LLU::DataList<LLU::NodeType::UTF8String>
instead of the generic data
list. The code would stay almost the same, except we could simply call node.value()
to get the corresponding std::string_view
.
For a different example, consider a function SeparateKeysAndValue
that takes a data store of named complex numbers and separates it into two data stores:
one holding names of the original data store and another one with values. For instance,
Developer`DataStore["a" -> 1 + 2.5 * I, "b" -> -3. - 6.I, 2I, "d" -> -4]
would be transformed into
Developer`DataStore[
"Keys" -> Developer`DataStore["a", "b", "", "d"],
"Values" -> Developer`DataStore[1. + 2.5 * I, -3. - 6.I, 2.I, -4.]
]
The implementation of such library function may look as follows:
EXTERN_C DLLEXPORT int SeparateKeysAndValues(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
LLU::MArgumentManager mngr(libData, Argc, Args, Res);
auto dsIn = mngr.getDataList<LLU::NodeType::Complex>(0);
DataList<LLU::NodeType::UTF8String> keys;
DataList<LLU::NodeType::Complex> values;
// we use structured bindings to immediately split each node into name and value
for (auto [name, value] : dsIn) {
keys.push_back(name);
values.push_back(value);
}
DataList<GenericDataList> dsOut; // the output type is a DataList of DataLists
dsOut.push_back("Keys", std::move(keys));
dsOut.push_back("Values", std::move(values));
mngr.set(dsOut);
}
Instead of a single loop with structured bindings, we could also make two passes over the input data list:
DataList<LLU::NodeType::UTF8String> keys;
for (auto name : LLU::NameAdaptor {dsIn}) {
keys.push_back(name);
}
DataList<LLU::NodeType::Complex> values;
for (auto value : LLU::ValueAdaptor {dsIn}) {
values.push_back(value);
}
Notice that we use iterator adaptors LLU::NameAdaptor
and LLU::ValueAdaptor
to iterate over a specific property of the nodes (either name or value,
respectively).
-
-
1$\begingroup$ Great answer! "The only problem is that for now it remains undocumented" That is indeed a big problem and I hope this will change soon. But this answer goes a long way to remedy the problem. Do you have any insights into performance compared to other possible methods I mentioned? $\endgroup$– SzabolcsJun 2, 2021 at 13:28
-
$\begingroup$ @Szabolcs I did a simple test once to compare the speed of raw DataStore, DataStore wrapper in LLU and WSTP wrapper in LLU, the code was sending 300k random strings to the library, reverting the order of characters in each string and then sending back to WL. WSTP turned out to be ~2.5x slower than DataStore. But this is far from proper benchmarking. My guess is that all workarounds based on serialization might be even slower. $\endgroup$– rafalcJun 2, 2021 at 16:31
-
2$\begingroup$ @Szabolcs DataStore is present in 11.0 but essentially it can only be used with AsynchronousTasks and not as an argument to LibraryLink functions. This functionality has been added in M12.0. $\endgroup$– rafalcJun 4, 2021 at 11:38
-
1$\begingroup$ @iago-lito almost correct, "DataStore" should be a String, so
LibraryFunctionLoad[lib, "fun_name", {"DataStore"}, "DataStore"]
$\endgroup$– rafalcJun 14, 2021 at 22:09
std::vector<std::string> mylist; mlink << mylist;
. $\endgroup$