17
$\begingroup$

When an application written in C++ depends on some libraries, the libraries are usually shipped together with the application. Two applications that depend on the same library will not share the same instance of that library. Compatibility problems are thus averted. Each application ships its own private instance of a known compatible (sometimes even patched) version of the library.

What is the best way to handle the same thing in Mathematica?

How should I design a package that is anticipated to be a dependency for multiple other packages?


Do we really need this? I think yes. LTemplate definitely needs this. What if I write a package whose only purpose is to handle persistent settings? It could (it is meant to) become a dependency for many others.


I do have a solution for this as I had to use it in LTemplate. I am asking this question because I think it will be useful to develop community standards. I would also like to see how others handle this problem and improve or simplify my own solution.


"What have you tried so far?"

Let's call the package SomePack. My requirements were the following:

  • Be able to load and use SomePack interactively with the usual Needs["SomePack`"]
  • Be able to embed SomePack into another package, and ship them together.
  • What if I want to load two packages each of which embed two incompatible versions of SomePack? When SomePack is embedded, it should have a unique context to avoid conflict with another instance embedded into another package.

Let's have the following directory structure:

SomePack 
  |
  |- Kernel
  |    |
  |    \-  init.m
  |
  |- SomePack.m
  |
  |- SomePackPrivate.m
  |
  \- SomePackInner.m

The files contain the following:

(* init.m *)
Get["SomePack`SomePack`"]
(* SomePack.m *)

BeginPackage["SomePack`"]
Get["SomePack`SomePackInner`"]
EndPackage[]
(* SomePackPrivate.m *)

(* Note the leading ` characters below!! *)
BeginPackage["`SomePack`"]
Quiet[ (* prevent false shadow warnings when loading SomePack first and OtherPack later *)
  Get["`SomePackInner`"], 
  General::shdw
] 
EndPackage[]
(* SomePackInner.m *)

SomeSymbol::usage = "SomeSymbol[]";

Begin["`Private`"]

(* code goes here *)

End[]

If the package is not embedded, load it with

Needs["SomePack`"]

Its context will be SomePack`

If it is embedded into OtherPack, then the SomePack directory should go within the OtherPack directory. Then OtherPack.m should follow this structure:

BeginPackage["OtherPack`"]

Get["`SomePack`SomePackPrivate`"]

Begin["`Private`"]

(* package code goes here *)

End[]
EndPackage[]

When loading OtherPack using Needs["OtherPack`"], SomePack will also get loaded and it will reside in OtherPack`SomePack`. Symbols from this context will be usable in the implementation of OtherPack, but they will not be public by default when loading OtherPack using Needs or Get.

$\endgroup$
  • 1
    $\begingroup$ I have thought about this problem quite a bit, when developing a package manager to be used with ProjectInstaller. Sadly, I never finished this. To put it short, in my opinion, the conventions like those you describe are a bit too complex to be easily used. I tried to solve the compatibility problem in a different way, by writing a custom loader that would assign namespace (context name) at run-time, so that you don't have to specify it explicitly at all. I think that this approach is potentially more powerful. $\endgroup$ – Leonid Shifrin Nov 19 '16 at 17:14
  • $\begingroup$ You can have a look at it in action here, where, while it isn't complete, I have a working prototype for such a loader, with an example. I was more concerned with the way to split a package into parts, but the same strategy could also be used for dependency management. Speaking of the latter, I had in mind to adopt the practices from npm and the node.js community, in how npm treats dependency management. I will try to specify the design as an answer, as soon as the time allows me to, hopefully within a day or two. $\endgroup$ – Leonid Shifrin Nov 19 '16 at 17:17
  • 1
    $\begingroup$ @Mr.Wizard I think we don't see this form the same perspective. There are multiple reasons: 1. I don't want to ask people to install package X to use my package. I want to bundle it. 2. When I say "version", I don't necessarily mean "released version 2.3.1". It could be that if my package Y depends on package X, I will just take the development version of X, even make a small necessary modification, then use it in Y. This X will be unique and different from any other version of X. How do I avoid conflicts with other X? I have to embed this X into my Y. What is a simple way to embed it? $\endgroup$ – Szabolcs Nov 20 '16 at 8:37
  • 2
    $\begingroup$ @Leonid I guess all this means that there really should be support for something like this directly from Wolfram. Mathematica has good support for interactive work, but package authors are a bit neglected ... $\endgroup$ – Szabolcs Nov 20 '16 at 12:48
  • 1
    $\begingroup$ A good example of loading, installing and patching a 3rd party package, all done automatically, is implemented in FeynCalc by @Rolf Mertig et al. FeynCalc loads,installs and patch FeynArts. $\endgroup$ – magma Dec 4 '16 at 11:59
4
$\begingroup$

The problem I see is that in principle every package should be suited to be embedded with specific context. And we can't expect everyone to start writing packages in this manner. So it should be OtherPackage` responsibility to do all required adjustments.

Moreover, you have to assume that someone in future will use OtherPackage` as an embedded package the same way as OtherPackage` uses SomePack`. So the solution used in OtherPackage` can't affect the standard package setup to much.

Here is my idea:

Assumptions

We need couple of assumptions.

  • all packages in a party are more or less written in a standard way

    That is BeginPackage -> Begin -> End -> EndPackage. Or at least their main Package.m file is written this way.

  • packages are written 'neatly', where contexts are managed by BeginPackage, Begin etc. In contrary to files written with full names e.g. var = SomePack`var2 ^ 2

I think this should be enough, is reasonable and, when doubt, can be verified easily by taking a look at the code.

SomePack` setup

Whatever fulfills assumptions.

OtherPack` setup

OtherPack
 |- Java
 |- libs
 |- WL
     |- SomePack (*it doesn't matter if it is here but I find it clean*)
 |- Kernel
 |- OtherPack.m
 |- PacletInfo.m

Fulfills assumptions and the OtherPack.m file has slightly modified way of loading dependencies

Begin["OtherPack`"]

   (*exported symbols*)

Internal`InheritedBlock[{BeginPackage, $Path},
   PrependTo[ (*1*)
      $Path, 
      FileNameJoin[{DirectoryName @ $InputFileName, "WL"}]
   ]; 

   Unprotect[BeginPackage]; 
     (*2 do this for all packages that need unique context *)
   BeginPackage["SomePack`"] = BeginPackage["SomePackForOtherPackage`"];

   Get /@ packagesInWLDir
];

Needs /@ globalDependencies (*JLink etc*)

Begin["Private`"]

...

End[];

EndPackage[];
  1. Packages setup often assumes that specific things are on $Path. E.g. in SomePackage` one may call <<SomePackage`Kernel`specificFile, this will break if SomePackage` is not on $Path. And if we have an embedded package then its directory probably won't be. Adding dependencies directory to $Path for loading makes this setup more general.

  2. Looks hairy but all it requires is SomePack` to be used in BeginPackage, which is recommended way anyway.

    One may want to modify this to catch BeginPackage[_, {___}] syntax too.

Possible problems

  • pregenerated GUIs

    If SomePack` contains Dialogs/Palettes that were created during deployment, then they contain full names with SomePack` context. Don't see an easy way to deal with this but it is not a common problem. Since such packages are meant to be used internally then you won't rely on their GUI anyway.

    It doesn't affect GUI generated on a fly from package code as they will be parsed with SomePackForOtherPack` context.

$\endgroup$
  • 1
    $\begingroup$ +1. This is a version of the custom loader that I mentioned in the comments. I am using similar scheme at work for a few projects, notably Streaming. That said, I think that the scheme should be a bit more complex than that., although based on similar ideas. If I happen to have the time to roll out a prototype quickly enough, will post it as a separate answer. $\endgroup$ – Leonid Shifrin Nov 20 '16 at 12:46
  • $\begingroup$ @LeonidShifrin I'm looing forward to it. I don't have a general solution for packages setup which support all features like this one at once but all my specific solutions are slowly converging to something useful. $\endgroup$ – Kuba Nov 20 '16 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.