# How to properly handle mutual imports of multiple packages?

I'm new to the mathematica platform and I am developing a simulation framework as my first exercise and have been tearing my hair figuring how to make packages interact with each other. Ill explain a minimum package experiment I developed to understand my problems and please I would like a theoretical explanation of why this happens, if it is a bug or if I am just dumb with mathematica.

The thing is that if I have packages that use each others exported symbols or functions inside their functions, after the first package is loaded he loads the others declared as needed as well, but then only that package that was loaded first can reference all other exported stuff, and the others for some reason can only look at the exported data of later packages/contexts in the context path. To make every package able to see every other exports I need to load them again individually. This makes no sense to me, I thought the whole point of having needs in any package constructor is to avoid loading myself every package of my packages set. Is this a bug? Btw, the exported symbols from different packages are found right away by mathematica is just the symbols inside functions that cant reference each other.

So basically I defined three packages, ja1,ja2,ja3. All have the same structure like this:

Package ja1

BeginPackage["SimulationsSystemja1", { "SimulationsSystemja2", "SimulationsSystemja3"}]

var1::usage = "var1"
pro1::usage = "pro1"
(* Exported symbols added here with SymbolName::usage *)

Begin["Private"] (* Begin Private Context *)

var1=1;
pro1[]:=var1+var2+var3

End[] (* End Private Context *)

EndPackage[]


Package ja2

BeginPackage["SimulationsSystemja2", { "SimulationsSystemja1", "SimulationsSystemja3"}]

var2::usage = "var2"
pro2::usage = "pro2"
(* Exported symbols added here with SymbolName::usage *)

Begin["Private"] (* Begin Private Context *)

var2=1;
pro2[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]


Package ja3

BeginPackage["SimulationsSystemja3", { "SimulationsSystemja1", "SimulationsSystemja2"}]

var3::usage = "var3"
pro3::usage = "pro3"
(* Exported symbols added here with SymbolName::usage *)

Begin["Private"] (* Begin Private Context *)

var3=1;
pro3[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]


Commands run: so after defining the packages I tried to run each pro function, called each var and called shadow to see the nearest Package/context working. And here is the result:

In[2]:= << SimulationsSystemja2
$ContextPath Out[5]= {"SimulationsSystemja2", "SimulationsSystemja1", \ "SimulationsSystemja3", "SimulationsSystemSimulationsSystem", \ "SimulationsSystemModelsBasic", \ "SimulationsSystemSimulFunctions", \ "SimulationsSystemTurnPackages", \ "SimulationsSystemEventFunctions", "SimulationsSystemClasses", \ "PacletManager", "WebServices", "System", "Global"} In[6]:= pro1[] Out[6]= 2 + SimulationsSystemja1Privatevar2 In[7]:= pro2[] Out[7]= 3 In[8]:= pro3[] Out[8]= 1 + SimulationsSystemja3Privatevar1 + \ SimulationsSystemja3Privatevar2 In[9]:= var1 Out[9]= 1 In[10]:= var2 Out[10]= 1 In[11]:= var3 Out[11]= 1 In[12]:= shadow Out[12]= 2 In[13]:= << SimulationsSystemja3$ContextPath

Out[14]= {"SimulationsSystemja3", "SimulationsSystemja2", \
"SimulationsSystemja1", "SimulationsSystemSimulationsSystem", \
"SimulationsSystemModelsBasic", \
"SimulationsSystemSimulFunctions", \
"SimulationsSystemTurnPackages", \
"SimulationsSystemEventFunctions", "SimulationsSystemClasses", \
"PacletManager", "WebServices", "System", "Global"}

In[15]:= pro1[]

Out[15]= 2 + SimulationsSystemja1Privatevar2

In[16]:= pro2[]

Out[16]= 3

In[17]:= pro3[]

Out[17]= 3

In[18]:= var1

Out[18]= 1

In[19]:= var2

Out[19]= 1

In[20]:= var3

Out[20]= 1

Out[21]= 3

In[22]:= << SimulationsSystemja1
$ContextPath Out[23]= {"SimulationsSystemja1", "SimulationsSystemja3", \ "SimulationsSystemja2", "SimulationsSystemSimulationsSystem", \ "SimulationsSystemModelsBasic", \ "SimulationsSystemSimulFunctions", \ "SimulationsSystemTurnPackages", \ "SimulationsSystemEventFunctions", "SimulationsSystemClasses", \ "PacletManager", "WebServices", "System", "Global"} In[24]:= pro1[] Out[24]= 3 In[25]:= pro2[] Out[25]= 3 In[26]:= pro3[] Out[26]= 3 In[27]:= var1 Out[27]= 1 In[28]:= var2 Out[28]= 1 In[29]:= var3 Out[29]= 1 In[30]:= shadow Out[30]= 1  You can see in the output that only after packages are loaded by themselves they are able to recognize every other packages' exported symbols and functions. - You put those packages in files in a path where they are all found by Needs or Get, right? Or you ran the code manually? – Rojo Sep 6 '12 at 21:41 the files are in a path and they can be found. I ran the packages from the workbench. I am sure they are found because normally if I make a mistake mathematica prints the get error, also the first package loaded pro function find the other packages symbol values. – Martin Perez Sep 6 '12 at 21:43 You seem to have gotten your “caps lock” key stuck, maybe that has something to do with it? – F'x Sep 6 '12 at 21:44 I dont have that key stuck, just tried to emphasize my conclusion – Martin Perez Sep 6 '12 at 21:48 I suppose you don't know: typing all in caps is considered very rude (equivalent to shouting). – Markus Roellig Sep 7 '12 at 19:33 show 1 more comment ## 2 Answers I don't have an access to Mathematica at the moment, so what follows is untested. What you observed can be understood by looking at the mechanics of package loading and symbols creation. I dealt with this problem before, and will re-post here verbatim my answer from the MathGroup thread: ### The setup and the problem Suppose you have two packages, the main one, and the helper one. The main one needs some of the functionality of the helper one, and you want to keep the context of the helper package on the context path after the main one loads. But, here is the problem: the helper one also needs some of the functionality of the main one. Such situation may be preferable for better designs in some rare cases. So, here is a naive attempt: (* Main package *) BeginPackage["MyMainPackage",{"MyHelperPackage"}] Fun1::usage="Fun1[x_] squares its argument"; Fun2::usage= "Fun2[x_] computes a square root of its argument"; Fun3::usage = "Fun3[x_,y_] is a more complex function"; Begin["Private"] (*Needs["MyHelperPackage"];*) Fun1[x_]:=x^2; Fun2[x_]:=Sqrt[x] Fun3[x_,y_]:=(x+y)*HFun1[x,y]; End[] EndPackage[] (* Helper package *) BeginPackage["MyHelperPackage"] HFun1::usage="This is a helper function"; Begin["Private"]; Needs["MyMainPackage"]; HFun1[x_,y_]:=Fun1[x]+Fun2[y] End[] EndPackage[]  Everything looks fine, so the function HFun1 ought to use functions Fun1 and Fun2 from the main package (this was what I was thinking anyway)). Now we try to use this: Needs["MyMainPackage"] HFun1[1, 2] (* MyHelperPackagePrivateFun1[1] + MyHelperPackagePrivateFun2[2] *) Fun3[1, 2] (* 3 (MyHelperPackagePrivateFun1[1] + MyHelperPackagePrivateFun2[2]) *)  Alas, this did not happen. The point is that the package MyHelperPackage is read before those definitions in the main package (because listing it in a list of dependent packages in BeginPackage["MyMainPackage",...] causes Needs to be called on MyHelperPackage before even the public part of the main package is read, and therefore, while the context "MyMainPackage" is on the contextpath of MyHelperPackage when the latter is read, the function names are not yet there in that context and could not be found. Therefore, MyHelperPackage made up its own private names for Fun1 and Fun2, which is obviously what we don't want. ### Possible solution: 1. If we don't need the context of the helper package to remain on the context path after the main package gets loaded, then there is no problem at all - import the helper package privately. In that case, the public portion of the main package with all public functions is read first, and this those names are already there by the time the helper package is loaded. Of course, the helper package must still import the main one (hidden import say). In code, this will now look like: BeginPackage["MyMainPackage"] Fun1::usage="Fun1[x_] squares its argument"; Fun2::usage= "Fun2[x_] computes a square root of its argument"; Fun3::usage = "Fun3[x_,y_] is a more complex function"; Begin["Private"] Needs["MyHelperPackage"]; Fun1[x_]:=x^2; Fun2[x_]:=Sqrt[x] Fun3[x_,y_]:=(x+y)*HFun1[x,y]; End[] EndPackage[]  Now we can check: Needs["MyMainPackage"] HFun1[1, 2] (* HFun1[1, 2] *) Fun3[1, 2] (* 3 (1 + Sqrt[2]) *)  We see that the problem has been solved in part, but that the helper context is then still unavailable: the HFun1[1, 2] evaluates to itself. 2. If we do need to leave the helper package on the context path (as in our example), this can be accomplished by the following trick: add lines BeginPackage["MyMainPackage",{"MyHelperPackage"}] EndPackage[]  to the end of your main package, which in its final form will look like: BeginPackage["MyMainPackage"] Fun1::usage="Fun1[x_] squares its argument"; Fun2::usage= "Fun2[x_] computes a square root of its argument"; Fun3::usage = "Fun3[x_,y_] is a more complex function"; Begin["Private"] Needs["MyHelperPackage"]; Fun1[x_]:=x^2; Fun2[x_]:=Sqrt[x] Fun3[x_,y_]:=(x+y)*HFun1[x,y]; End[] EndPackage[] BeginPackage["MyMainPackage",{"MyHelperPackage"}] EndPackage[]  In this case, starting with a fresh kernel, everything works as planned: Needs["MyMainPackage"] HFun1[1, 2] (* 1 + Sqrt[2] *) Fun3[1, 2] (* 3 (1 + Sqrt[2]) *)  ### Remark Now, one may argue that mutual imports may reflect bad design, but IMO they can, very occasionally, be quite handy and lead to actually better designs. In any case, this is another possibility, whether or not to use it is a different issue. - Any difference between appending the BeginPackage vs appending a Needs? – Rojo Sep 7 '12 at 0:01 Yeah the needs do not keep the package context in the context path. – Martin Perez Sep 7 '12 at 3:24 @MartinPerez I don't think it is your OOP background, but sometimes it is unclear how to separate them. This can lead to circular dependencies (mutual dependence has a circle of size 2). Per Leonid's example, I would consider removing the dependency HFun1 on Fun1 and Fun2 by having them passed as parameters. Or, have settable "constants" like $PreRead. Since functions can be passed like any other variable, I'd consider something like this to break your dependencies. – rcollyer Sep 7 '12 at 3:41
@Rojo Needs, not only its context will be added again to the $ContextPath, but also the contexts of all those packages it was importing via the second argument of BeginPackage. At some point, it took me a while to decipher this behavior, since it looks like you call Needs on one context and get several contexts added to the $ContextPath. Of course, this seems natural in retrospect, but this is still something worth knowing. – Leonid Shifrin Sep 7 '12 at 9:24
@MartinPerez My experience is that in Mathematica, mutual imports indicate insufficiently thought out design most of the time. However, there may be rare cases where they might be justified. The usual arguments apply: minimize coupling, maximize coherence. – Leonid Shifrin Sep 7 '12 at 9:26

When implementing Leonid possible solution in the second case (needing contexts to stay in context path) found an important inconvenience (at least for me) when working with the workbench. It is that any symbol in the middle of code belonging to a package that was not included as needed in BeginPackage[] would not be colored GRAY (default color for public/imported definitions) and would not give you the USAGE info. So I think I found out an even better solution that also handles mutual import over many simultaneos packages uniformly and keeps the workbench export utility working. But this is not the perfect solution yet, still have to move above the export block the needs block after exporting symbols

Solution:

The new structure for any package would be

1.-BeginPackage["mainpackage"]

2.-public definitions with usage (its important that all definitions stay before the needs)

3.-needs["helperpackage"] for all needed packages

4.-EndPackage[]

5.-BeginPackage["mainpackage",{"helper1","helper2",...}] (no code between this beginpackage and the begin private statement)

6.-Begin["private"]

7.-The rest as usual

Like this employing my previous example with ja1,ja2,ja3:

Package JA1

BeginPackage["SimulationsSystemja1"]
var1::usage = "var1"
pro1::usage = "pro1"
(* Exported symbols added here with SymbolName::usage *)
Needs["SimulationsSystemja2"]
Needs["SimulationsSystemja3"]
EndPackage[]

BeginPackage["SimulationsSystemja1", { "SimulationsSystemja2", "SimulationsSystemja3"}]
Begin["Private"] (* Begin Private Context *)
var1=1;
pro1[]:=var1+var2+var3

End[] (* End Private Context *)

EndPackage[]


Package JA2

BeginPackage["SimulationsSystemja2"]

var2::usage = "var2"
pro2::usage = "pro2"
(* Exported symbols added here with SymbolName::usage *)
Needs["SimulationsSystemja1"]
Needs["SimulationsSystemja3"]
EndPackage[]

BeginPackage["SimulationsSystemja2", { "SimulationsSystemja1", "SimulationsSystemja3"}]
Begin["Private"] (* Begin Private Context *)
Needs["SimulationsSystemja3"]
var2=1;
pro2[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]


Package JA3

BeginPackage["SimulationsSystemja3"]

var3::usage = "var3"
pro3::usage = "pro3"
(* Exported symbols added here with SymbolName::usage *)
Needs["SimulationsSystemja1"]
Needs["SimulationsSystemja2"]

pro5::usage = "pro5  "
EndPackage[]

BeginPackage["SimulationsSystemja3", { "SimulationsSystemja1", "SimulationsSystemja2"}]
Begin["Private"] (* Begin Private Context *)
var3=1;
pro3[]:=var1+var2+var3
pro5[]:=a
End[] (* End Private Context *)

EndPackage[]


Output:

In[1]:= << SimulationsSystemja1
$ContextPath During evaluation of In[1]:= shadow::shdw: Symbol shadow appears in multiple contexts {SimulationsSystemja2,SimulationsSystemja1}; definitions in context SimulationsSystemja2 may shadow or be shadowed by other definitions. >> During evaluation of In[1]:= shadow::shdw: Symbol shadow appears in multiple contexts {SimulationsSystemja3,SimulationsSystemja2,SimulationsSystemja1}; definitions in context SimulationsSystemja3 may shadow or be shadowed by other definitions. >> Out[1]= {"SimulationsSystemja1", "SimulationsSystemja2", \ "SimulationsSystemja3", "PacletManager", "WebServices", \ "System", "Global"} In[2]:= pro1[] Out[2]= 3 In[3]:= pro2[] Out[3]= 3 In[4]:= pro3[] Out[4]= 3 In[5]:= var1 Out[5]= 1 In[6]:= var2 Out[6]= 1 In[7]:= var3 Out[7]= 1 In[8]:= shadow Out[8]= 1  It Works! - +1 - looks like a good alternative. – Leonid Shifrin Sep 7 '12 at 9:27 Really? Why do we need these tricks? I think the whole Package setup is not nice and completely oldfashioned and there should be something better. I wished these fundamental problems in software design would be addressed in some version of Mathematica. However, it has not changed much since 1987 or so. – Rolf Mertig Sep 7 '12 at 9:37 @RolfMertig I disagree with you this once, Rolf. I think the package system is very well designed and thought out, in that the user gets enormous control over all encapsulation mechanisms, most of which are exposed to us. The problem under discussion I view a consequence of this. Should we have a more rigid packaging system, it might have addressed this one, but it would restrict what one can possibly do. I do agree that such a system might be needed, but it can as well be implemented on top of the existing one (which, indeed, has not been done yet). – Leonid Shifrin Sep 7 '12 at 9:43 @LeonidShifrin Sure, one can control everything, and I and you can (though I basically end up using full context names a lot), and one can manipulate$Path and alike. But try to explain this to someone new. And that "Workbench" does not help at all with subpackages. It is really really not enough for non-specialists. And even I get annoyed all the time when things get slightly more involved. There are subtle issues with SetDelayed and contexts, also circular Needs can be tricky. And by now I gave up with DeclarePackage. It is just too cumbersome to get it to do what one wants. – Rolf Mertig Sep 7 '12 at 10:36
@RolfMertig I would use Block and modify locally the relevant variables ($ContextPath, $Packages and $Context), plus modify globally some of them too ($ContextPath, $Packages). Can't offer a complete solution right now, will think about it, but it does not seem to be too hard. The key point is that the parsing (creation of symbols in the correct context) is completely controlled by the current values of $Context and \$ContextPath`, which both are exposed to the user. – Leonid Shifrin Sep 7 '12 at 17:42