Strategies and practical advice for using saved compiled functions in a package

Question

I want to use a compiled function in a package without having to rebuild it each time. This slows down unit testing and loading of the package.

I have used the example in the link as my starting point: Exporting a compiled function for later use

Briefly

1. Execute func=Compile[...] in a notebook

2. Save the .dll funcLoc = func[[-1, 1]] in the package location

3. Save func as originalFuncDefinition.mx with DumpSave

4. In the package do Get["originalFuncDefinition"]; to expose the function symbol.

5. In the package redefine func as

PackageExport["func"]
func = ReplacePart[func,{-1, 1} -> funcLoc ];

Problems I feel I have with my approach

1. I write packages in an IDE and steps 1-3 being in a notebook are not available to IDE for editing / refactoring making debugging difficult down the line.
2. Steps 1 to 3 are difficult to automate as if you have several functions, you need to DumpSave the symbols one by one eg DumpSave[{func1, func2 ...}] you cannot pass it as a list eg funcs = {func1,func2,...}; DumpSave/@funcs
3. Step 4 exposes all the original definitions globally...
4. ...So I am not always sure which func is being used
5. This involves working back and forth between the notebook and the package making sure that funcLoc tallies to each func making it prone to error

Any advice, tips or ways to think about this would be appreciated.

Answer 1

This is my strategy which solves all my problems (but not in the most elegant way, and I feel perhaps there is a better method? Please feel free to chime in)

This also solves another problem I had: sharing code between different computers which had different path names.

---

Broadly I create an Association with the function names defined in strings, this way I don't have any symbol clashes. These string names point to the compiled code.

I save these details as compiledFunctionDetails with the function storeCompiledFunctionDetails. This function also creates the DLL files for later.

These details also Hold your file path so it doesn't evaluate and you can use them on other PCs assuming your path is similar eg: FileNameJoin@{$UserDocumentsDirectory, "Wolfram Mathematica", "myDLLs"}

I then use a helper function readCompiledFunctionDetails to call the already compiled code, releasing the held filename. Crucially, the DLLs is referenced by its string name so we can now assign its symbol name to it for the first time.

eg: f1C = LibraryFunctionLoad @@ readCompiledFunctionDetails["f1C"]

---

Here is the code

(* 1. Define compiled functions here, <|"functionName"->Compile[etc.],...|> *)
compiledFunctions =
 <|
  "f1C" ->
   Compile[
    {
     {a, _Real}, {b, _Real}
     }
    ,
    a + b
    , Parallelization -> True,
    CompilationTarget -> "C",
    RuntimeOptions -> "Speed",
    RuntimeAttributes -> Listable
    ],
  "f2C" ->
   Compile[
    {
     {a, _Real}, {b, _Real}
     }
    ,
    a*b
    , Parallelization -> True,
    CompilationTarget -> "C",
    RuntimeOptions -> "Speed",
    RuntimeAttributes -> Listable
    ]
  |>

(* 2. Path to store DLLs and the config file: compiledFunctionDetails *)
(* *** YOUR FOLDER HERE *** *)
myDLLfolder = Hold[FileNameJoin@{$UserDocumentsDirectory, "Wolfram Mathematica","myDLLs"}];


(* 3. Function to save DLLs *)
storeCompiledFunctionDetails[cfName_String, cf_CompiledFunction] :=
  With[
   {
    cfList = (cf /. {CompiledFunction -> List, LibraryFunction -> List}),
    compiledFunctionsDllsPath = myDLLfolder
    },
   Module[
    {
     dllTemporaryFileName = Last[cfList][[1]],
     cfNameAndSignature = Last[cfList][[2 ;;]],
     libraryFunctionArguments, dllFileBaseName, dllFileName, dllFileNameNew, dllHeldNameToRead
     },
    
    (* copy dll locally *)
    dllFileBaseName = FileNameTake[dllTemporaryFileName, -1];
    dllFileName = FileNameJoin[{ReleaseHold@compiledFunctionsDllsPath,dllFileBaseName}]; (* unhold to copy *)
    CopyFile[dllTemporaryFileName, dllFileName, OverwriteTarget -> True];
    
    (* rename *)
    dllFileNameNew = FileNameJoin@{ReleaseHold@compiledFunctionsDllsPath, cfName <> ".dll"}; (* unhold to rename *)
    RenameFile[dllFileName, dllFileNameNew, OverwriteTarget -> True];
    
    (* instructions for LibraryFunctionLoad *)
    dllHeldNameToRead = FileNameJoin@{compiledFunctionsDllsPath, cfName <> ".dll"}; (* hold so is works on other computers *)
    libraryFunctionArguments = Join[{dllHeldNameToRead}, cfNameAndSignature];
    <|cfName -> libraryFunctionArguments|>
    ]
   ];


(* 4. Function to read DLLs *)
compiledFunctionDetailsFileName = FileNameJoin[{ReleaseHold[myDLLfolder], "CompiledFunctionDetails.mx"}];
readCompiledFunctionDetails = Map[ReleaseHold, Import[compiledFunctionDetailsFileName], {2}];


(* 5. Export the compiled code *)
compiledFunctionDetails = Association[ KeyValueMap[storeCompiledFunctionDetails, compiledFunctions]]

(* 6. Export instruction the config file *)
Export[compiledFunctionDetailsFileName, compiledFunctionDetails];

---

(* Now place this in your paclet / package or elsewhere *)
(* Load the function *)
f1C = LibraryFunctionLoad @@ readCompiledFunctionDetails["f1C"]
f2C = LibraryFunctionLoad @@ readCompiledFunctionDetails["f2C"]

f1C[40, 2] (* 42. *)
f2C[40, 2] (* 80. *)