# How can I use shared libraries in LibraryLink code and ensure Mathematica will find them?

I need to use shared libraries in some LibraryLink code. The shared library is installed in a custom location because it was also compiled by myself. When I try to load functions from the LibraryLink library, I get LibraryFunction::libload errors because Mathematica is not able to find the shared library.

What is the solution to this problem?

-
This is not really a Mathematica question, but it is one which is likely to come up during LibraryLink development. –  Szabolcs Oct 14 '13 at 19:56

## 2 Answers

Let me describe a way which works through all systems and simplifies the distribution of library code within a package a lot.

First I want to point out that there are two major scenarios here:

1. You are currently developing a package containing LibraryLink functions. When you are actively working on such a package, it is most likely not installed in your $UserAddOnsDirectory but instead in your development path. Nevertheless, you want to test, debug or run your library functions within Mathematica. On those development machines I find it quite convenient to either set the Mathematica $LibraryPath variable or provide the appropriate environment variables to Mathematica. The latter can be annoying sometimes, especially on OSX because they change the way of sending env-variables to applications every five minutes.

2. The second scenario is when you have finished your package development and it is ready to be installed (or distributed to other machines). Here, there is a much easier way to solve the dependency problem which I will describe in the following

### Set up your package structure correctly

Your package should have the library functions and dependency libraries in the appropriate folder under Package/LibraryResources/$SystemID. As example, here the directory structure of one of my projects IPCU ├── Documentation │ └── English │ └── ReferencePages │ └── Symbols ├── Java ├── Kernel └── LibraryResources ├── Linux-x86-64 ├── MacOSX-x86-64 ├── Windows └── Windows-x86-64  The overall advantage of putting a package structured like above into a place where Mathematica can find it is that it will automatically add the correct library path to its $LibraryPath variable and all .so|.dylib|.dll will be found.

### Loading the dependencies before your LibraryFunction

Now you can first check that all your libraries can be found even if you haven't set any environment variable explicitly or hard-coded the library position like in Szabolcs answer. To give an example for my package:

(* on Linux *)
FindLibrary["libtbb"]
(* /home/patrick/.Mathematica/Applications/IPCU/LibraryResources/Linux-x86-64/libtbb.so *)

(* on OSX *)
FindLibrary["libtbb"]
(* /Users/patrick/Library/Mathematica/Applications/\
IPCU/LibraryResources/MacOSX-x86-64/libtbb.dylib *)


Now, what you have to do is to load all dependency libraries before you load you actual LibraryLink functions. Please be careful, because order might be important. If you get an error, please check the content of LibraryLink$LibraryError, which gives a more detailed reason for failure. To grab an example, the underlying parallel LibraryLink function from my answer here uses the above mentioned Intel TBB library. If I just try to load the calculateNewtonFractal LibraryLink function, I get the following error LibraryFunction::libload: The function calculateNewtonFractal was not loaded from the file /home/patrick/.Mathematica/Applications/IPCU/LibraryResources/Linux-x86-64/ipcu_library.so. and the content of LibraryLink$LibraryError is

"libtbb.so.2: cannot open shared object file: No such file or directory"

If I instead preload the tbb, it works excellent:

LibraryLoad["libtbb"];
cNewtonFractalShared=LibraryFunctionLoad[
"ipcu_library","calculateNewtonFractal",
{{_Complex,1,"Constant"},{_Real,1,"Constant"},{_Real,1,"Constant"},
{_Real,0,"Constant"},{_Integer,3,"Shared"}},"Void"]

(* LibraryFunction[<>,calculateNewtonFractal,{{Complex,1,Constant},
{Real,1,Constant},{Real,1,Constant},{Real,0,Constant},{Integer,3,Shared}},{}] *)


### Conclusion

Therefore, the condensed required steps are:

1. Make sure your dependency libraries can be found with FindLibrary by (i) using above package structure, (ii) setting $LibraryPath manually or (iii) setting system environment variables and providing them to Mathematica. 2. Load the dependency libraries in the correct order (they might depend on each other) using LibraryLoad. Check LibraryLink$LibraryError if you get errors.
3. Load your LibraryLink function using LibraryFunctionLoad.
-
Good points, +1. With RLink, I was actually solving a different problem - how to make some shared library (R'd core library) find its dependencies, when called from within Mathematica (actually loaded into the JVM of JLink). So I had no control over the structure of the R project, short of writing my own build script for R (which is a daunting task). –  Leonid Shifrin Oct 15 '13 at 7:23

On OS X the usual solution is to set the DYLD_LIBRARY_LINK environment variable to specify custom search paths for shared libraries. This is quite inconvenient when Mathematica is launched as a GUI program, but there is an alternative solution using install names:

It is possible to hard code the install location of a shared library into the library itself. When an executable that uses this shared library is linked, this location will be copied into it. If the library doesn't already have this information encoded, it is possible to add it using the following command in the terminal:

install_name_tool -id full/path/to/lib/to/load libfile.dylib


After this is done, all we need to do is recompile the LibraryLink code normally, using CreateLibrary`, and it will know where to look for its shared libraries.

-
This is one possible Mac-only solution. Others will hopefully post solutions for other platforms, or better Mac-specific solutions. –  Szabolcs Oct 14 '13 at 19:56
+1. This was the basis of how I made RLink which comes with Mathematica to be found by it. I had actually to write a Mathematica script which first tracked all dependencies and then patched a number of libraries in the R distro. Similar thing can be achieved in Linux by using the patchelf utility. In fact, I do like the Linux mechanism better, since on Mac the shared lib tends to hard-code the path to the lib on which it depends, upon the first use. On Windows, it is also possible, and actually simpler, since one can modify the Path variable more or less at run-time. –  Leonid Shifrin Oct 14 '13 at 23:02
When I get some time, I can post the solutions for other platforms, and expand a bit on the one for Mac that you described. –  Leonid Shifrin Oct 14 '13 at 23:03
On Windows, one would typically write manifest files to accomplish the same goal. This is a somewhat involved matter, though: see e.g. here. Microsoft's solution to "DLL hell" seems rather cumbersome to say the least, but on the other hand it is more secure than altering environment variables. –  Oleksandr R. Oct 15 '13 at 18:58