This cannot be answered very well without knowing your C++ library much better.
As you said, you have a choice between MathLink and LibraryLink. Generally, I recommend LibraryLink because:
- It runs in the same process, and data transfer is much faster than with MathLink
- It provides features that MathLink does not have, such as direct manipulation of packed arrays, managed library expressions, fast callback functions
- You can use MathLink for communication in LibraryLink functions when you need it
MathLink is not really an extension API. It is a communication protocol which works either within a single process, between two processes (typical way), or even through the network.
The two main advantages of extending Mathematica through installable MathLink programs (instead of LibraryLink):
Easier to get started because much of the interface code is generated for you by the mprep utility from .tm template files. See tutorial.
Your code will run in a separate process, and will communicate with Mathematica through the MathLink protocol. If your code crashes, it won't take down the whole Mathematica kernel. (This is what happens when it crashes.)
I use LibraryLink a lot myself, and created the LTemplate package to make it just as easy to use as MathLink template files:
Just like mprep, this package will generate much of the interface code from a "template" (this time given as a Mathematica expression).
A big difference from MathLink template files is that LTemplate relies on C++ heavily (C is not sufficient), and the templates describe an interface to a C++ class. Thus you must work with classes. On the upside, it automates creating managed library expressions, which in plain term means that you won't have to delete your objects explicitly from Mathematica—it will be done for you when there are no more references to them.
So far it sounds like a perfect match for your project, and I do recommend that you should check it out.
But be aware that:
You will not be able to just write a template for your existing C++ classes (unless they are trivial). You will need to create a wrapper class that is appropriate for LTemplate and uses its types (which correspond to Mathematica's types)
You really should not expose a class interface to your users in Mathematica. Mathematica is not Python, and it does not like mutable state. I strongly recommend to hide the fact that you are using classes in the background, and write a simpler function interface that avoids mutable state completely. This is what I do in my IGraph/M package, which is based on LTemplate. When I write something for my own private use, I often leave the classes exposed because I know how to use them. But doing this will likely confuse many Mathematica users, who are not used to this. (To be fair, I should point out that Mathematica ships with some packages which expose such an interface, and require manual object management: TriangleLink and TetGenLink.)
LTemplate is still a work in progress. Since I did not get a lot of feedback about it, I only worked on features I needed myself. It currently supports integer, real and complex dense and sparse arrays, as well as the typical scalar types (boolean, string, number). It's what most math-focused libraries need. It also lets you use MathLink-based communication when you need it.
RawArray support is in the works, and will be ready soon. RawArray is an undocumented type that you probably don't need. Support for Image (MImage type) and better integration with C++ templates is also in the works, but this won't be ready anytime soon.
