10
$\begingroup$

I'm new to the mathematica platform and I am developing a simulation framework as my first exercise and have been tearing my hair figuring how to make packages interact with each other. Ill explain a minimum package experiment I developed to understand my problems and please I would like a theoretical explanation of why this happens, if it is a bug or if I am just dumb with mathematica.

The thing is that if I have packages that use each others exported symbols or functions inside their functions, after the first package is loaded he loads the others declared as needed as well, but then only that package that was loaded first can reference all other exported stuff, and the others for some reason can only look at the exported data of later packages/contexts in the context path. To make every package able to see every other exports I need to load them again individually. This makes no sense to me, I thought the whole point of having needs in any package constructor is to avoid loading myself every package of my packages set. Is this a bug? Btw, the exported symbols from different packages are found right away by mathematica is just the symbols inside functions that cant reference each other.

So basically I defined three packages, ja1,ja2,ja3. All have the same structure like this:

Package ja1

BeginPackage["SimulationsSystem`ja1`", { "SimulationsSystem`ja2`", "SimulationsSystem`ja3`"}]

shadow::usage="shadow"
var1::usage = "var1"
pro1::usage = "pro1"
(* Exported symbols added here with SymbolName::usage *)  

Begin["`Private`"] (* Begin Private Context *) 

shadow=1;
var1=1;
pro1[]:=var1+var2+var3

End[] (* End Private Context *)

EndPackage[]

Package ja2

BeginPackage["SimulationsSystem`ja2`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja3`"}]

shadow::usage="shadow"
var2::usage = "var2"
pro2::usage = "pro2"
(* Exported symbols added here with SymbolName::usage *)  

Begin["`Private`"] (* Begin Private Context *) 

shadow=2;
var2=1;
pro2[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]

Package ja3

BeginPackage["SimulationsSystem`ja3`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja2`"}]

shadow::usage="shadow"
var3::usage = "var3"
pro3::usage = "pro3"
(* Exported symbols added here with SymbolName::usage *)  

Begin["`Private`"] (* Begin Private Context *) 

shadow=3;
var3=1;
pro3[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]

Commands run: so after defining the packages I tried to run each pro function, called each var and called shadow to see the nearest Package/context working. And here is the result:

In[2]:= << SimulationsSystem`ja2`
$ContextPath

Out[5]= {"SimulationsSystem`ja2`", "SimulationsSystem`ja1`", \
"SimulationsSystem`ja3`", "SimulationsSystem`SimulationsSystem`", \
"SimulationsSystem`Models`Basic`", \
"SimulationsSystem`SimulFunctions`", \
"SimulationsSystem`TurnPackages`", \
"SimulationsSystem`EventFunctions`", "SimulationsSystem`Classes`", \
"PacletManager`", "WebServices`", "System`", "Global`"}

In[6]:= pro1[]

Out[6]= 2 + SimulationsSystem`ja1`Private`var2

In[7]:= pro2[]

Out[7]= 3

In[8]:= pro3[]

Out[8]= 1 + SimulationsSystem`ja3`Private`var1 + \
SimulationsSystem`ja3`Private`var2

In[9]:= var1

Out[9]= 1

In[10]:= var2

Out[10]= 1

In[11]:= var3

Out[11]= 1

In[12]:= shadow

Out[12]= 2

In[13]:= << SimulationsSystem`ja3`
$ContextPath

Out[14]= {"SimulationsSystem`ja3`", "SimulationsSystem`ja2`", \
"SimulationsSystem`ja1`", "SimulationsSystem`SimulationsSystem`", \
"SimulationsSystem`Models`Basic`", \
"SimulationsSystem`SimulFunctions`", \
"SimulationsSystem`TurnPackages`", \
"SimulationsSystem`EventFunctions`", "SimulationsSystem`Classes`", \
"PacletManager`", "WebServices`", "System`", "Global`"}

In[15]:= pro1[]

Out[15]= 2 + SimulationsSystem`ja1`Private`var2

In[16]:= pro2[]

Out[16]= 3

In[17]:= pro3[]

Out[17]= 3

In[18]:= var1

Out[18]= 1

In[19]:= var2

Out[19]= 1

In[20]:= var3

Out[20]= 1

In[21]:= shadow

Out[21]= 3

In[22]:= << SimulationsSystem`ja1`
$ContextPath

Out[23]= {"SimulationsSystem`ja1`", "SimulationsSystem`ja3`", \
"SimulationsSystem`ja2`", "SimulationsSystem`SimulationsSystem`", \
"SimulationsSystem`Models`Basic`", \
"SimulationsSystem`SimulFunctions`", \
"SimulationsSystem`TurnPackages`", \
"SimulationsSystem`EventFunctions`", "SimulationsSystem`Classes`", \
"PacletManager`", "WebServices`", "System`", "Global`"}

In[24]:= pro1[]

Out[24]= 3

In[25]:= pro2[]

Out[25]= 3

In[26]:= pro3[]

Out[26]= 3

In[27]:= var1

Out[27]= 1

In[28]:= var2

Out[28]= 1

In[29]:= var3

Out[29]= 1

In[30]:= shadow

Out[30]= 1

You can see in the output that only after packages are loaded by themselves they are able to recognize every other packages' exported symbols and functions.

$\endgroup$
6
  • $\begingroup$ You put those packages in files in a path where they are all found by Needs or Get, right? Or you ran the code manually? $\endgroup$
    – Rojo
    Sep 6, 2012 at 21:41
  • $\begingroup$ the files are in a path and they can be found. I ran the packages from the workbench. I am sure they are found because normally if I make a mistake mathematica prints the get error, also the first package loaded pro function find the other packages symbol values. $\endgroup$ Sep 6, 2012 at 21:43
  • $\begingroup$ I dont have that key stuck, just tried to emphasize my conclusion $\endgroup$ Sep 6, 2012 at 21:48
  • 1
    $\begingroup$ I suppose you don't know: typing all in caps is considered very rude (equivalent to shouting). $\endgroup$ Sep 7, 2012 at 19:33
  • $\begingroup$ @MarkusRoellig understood Markus, I changed that. $\endgroup$ Sep 7, 2012 at 20:54

2 Answers 2

14
$\begingroup$

General

First of all, let me say that the mutual dependency of this type is usually a sign of sub-optimal design - if your two functions need to access each other, then it may mean that they should actually belong to the same package. However, in some rare cases such design can indeed simplify things.

What you observed can be understood by looking at the mechanics of package loading and symbols creation. I dealt with this problem before, and will re-post here almost verbatim my answer from the MathGroup thread, and extend it a little:

Mutual import of packages in the standard package format

The problem

Suppose you have two packages, the main one, and the helper one. The main one needs some of the functionality of the helper one, and you want to keep the context of the helper package on the context path after the main one loads. But, here is the problem: the helper one also needs some of the functionality of the main one. Such situation may be preferable for better designs in some rare cases.

So, here is a naive attempt:

(* Main package *)

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

and a helper (second) package

(* Helper package *)

BeginPackage["MyHelperPackage`"]

HFun1::usage="This is a helper function";

Begin["`Private`"];

Needs["MyMainPackage`"];

HFun1[x_,y_]:=Fun1[x]+Fun2[y]

End[]
EndPackage[]

Everything looks fine, so the function HFun1 ought to use functions Fun1 and Fun2 from the main package (this was what I was thinking anyway)). Now we try to use this:

Needs["MyMainPackage`"]

HFun1[1, 2] 

(* MyHelperPackage`Private`Fun1[1] + MyHelperPackage`Private`Fun2[2] *)

Fun3[1, 2]

(* 3 (MyHelperPackage`Private`Fun1[1] + MyHelperPackage`Private`Fun2[2]) *)

Alas, what we hoped for did not happen. The point is that the package MyHelperPackage` is read before those definitions in the main package (because listing in in a list of dependent packages in BeginPackage["MyMainPackage`",...] causes Needs to be called on MyHelperPackage` before even the public part of the main package is read, and therefore, while the context "MyMainPackage`" is on the contextpath of MyHelperPackage`, when the latter is read, the function names are not yet there in that context and could not be found. Therefore, MyHelperPackage` made up its own private names for Fun1 and Fun2, which is obviously what we don't want.

Solution #1: hidden import of one of the packages

If we don't need the context of the helper package to remain on the contextpath after the main package gets loaded, then there is no problem at all - import the helper package privately. In that case, the public portion of the main package with all public functions is read first, and this those names are already there by the time the helper package is loaded. Of course, the helper package must still import the main one (hidden import say). In code, this will now look like:

BeginPackage["MyMainPackage`"]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Needs["MyHelperPackage`"];

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

Now we can check:

Needs["MyMainPackage`"]

HFun1[1, 2] 

(* HFun1[1, 2] *)

Fun3[1, 2]

(* 3 (1 + Sqrt[2]) *)

We see that the problem has been solved in part, but that the helper context is then still unavailable: the HFun1[1, 2] evaluates to itself.

Solution 2: using a trick with idle BeginPackage - EndPackage

If we do need to leave the helper package on the context path (as in our example), this can be accomplished by the following trick: add lines

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]
EndPackage[]

to the end of your main package, which in its final form will look like:

BeginPackage["MyMainPackage`"]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Needs["MyHelperPackage`"];

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]
EndPackage[]

In this case, starting with a fresh kernel, everything works as planned:

Needs["MyMainPackage`"]

HFun1[1, 2]

(* 1 + Sqrt[2] *)

Fun3[1, 2]

(* 3 (1 + Sqrt[2]) *)

Solution #3: Use fully-qualified symbol names, in one of the packages

This solution is perhaps not as good as the other ones.The bad thing about it is that fully-qualified symbols couple together the two packages (the one from where they come and the one where they are used), more tightly than it may be desirable. This may lead to various problems, such as regressions due to refactorings, where a function is moved to a different package, or entire package's context is changed. However, in practice this one is often used.

So, the main package would look now just like in our initial naive attempt:

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

and I decided to use fully qualified names for symbols from the main package used in a helper one (which may make more sense than otherwise, since main package's functions may represent more stable public API):

(* Helper package *)

BeginPackage["MyHelperPackage`"]

HFun1::usage="This is a helper function";

Begin["`Private`"];

HFun1[x_,y_]:=MyMainPackage`Fun1[x]+MyMainPackage`Fun2[y]

End[]
EndPackage[]

Depending on the situation, this version may be a version of choice (e.g. you know for sure that the symbol long names are not going to change etc.), but still I'd avoid this in favor of one of the other mentioned solutions.

Summary

I have presented three different ways to solve the symbol resolution issue for mutual package imports. I think that one should be able to use one of the suggested solutions successfully in the majority of cases.

Now, one may argue that mutual imports may reflect bad design, but IMO they can, very occasionally, be quite handy and lead to actually better designs.

$\endgroup$
15
  • 3
    $\begingroup$ Any difference between appending the BeginPackage vs appending a Needs? $\endgroup$
    – Rojo
    Sep 7, 2012 at 0:01
  • 1
    $\begingroup$ Yeah the needs do not keep the package context in the context path. $\endgroup$ Sep 7, 2012 at 3:24
  • 1
    $\begingroup$ @MartinPerez I don't think it is your OOP background, but sometimes it is unclear how to separate them. This can lead to circular dependencies (mutual dependence has a circle of size 2). Per Leonid's example, I would consider removing the dependency HFun1 on Fun1 and Fun2 by having them passed as parameters. Or, have settable "constants" like $PreRead. Since functions can be passed like any other variable, I'd consider something like this to break your dependencies. $\endgroup$
    – rcollyer
    Sep 7, 2012 at 3:41
  • 1
    $\begingroup$ @Rojo Needs, not only its context will be added again to the $ContextPath, but also the contexts of all those packages it was importing via the second argument of BeginPackage. At some point, it took me a while to decipher this behavior, since it looks like you call Needs on one context and get several contexts added to the $ContextPath. Of course, this seems natural in retrospect, but this is still something worth knowing. $\endgroup$ Sep 7, 2012 at 9:24
  • 1
    $\begingroup$ @MartinPerez My experience is that in Mathematica, mutual imports indicate insufficiently thought out design most of the time. However, there may be rare cases where they might be justified. The usual arguments apply: minimize coupling, maximize coherence. $\endgroup$ Sep 7, 2012 at 9:26
3
$\begingroup$

When implementing Leonid possible solution in the second case (needing contexts to stay in context path) found an important inconvenience (at least for me) when working with the workbench. It is that any symbol in the middle of code belonging to a package that was not included as needed in BeginPackage[] would not be colored GRAY (default color for public/imported definitions) and would not give you the USAGE info. So I think I found out an even better solution that also handles mutual import over many simultaneos packages uniformly and keeps the workbench export utility working. But this is not the perfect solution yet, still have to move above the export block the needs block after exporting symbols

Solution:

The new structure for any package would be

1.-BeginPackage["mainpackage"]

2.-public definitions with usage (its important that all definitions stay before the needs)

3.-needs["helperpackage"] for all needed packages

4.-EndPackage[]

5.-BeginPackage["mainpackage",{"helper1","helper2",...}] (no code between this beginpackage and the begin private statement)

6.-Begin["private"]

7.-The rest as usual

Like this employing my previous example with ja1,ja2,ja3:

Package JA1

BeginPackage["SimulationsSystem`ja1`"]
shadow::usage="shadow"
var1::usage = "var1"
pro1::usage = "pro1"
(* Exported symbols added here with SymbolName::usage *)
Needs["SimulationsSystem`ja2`"]
Needs["SimulationsSystem`ja3`"]  
EndPackage[]

BeginPackage["SimulationsSystem`ja1`", { "SimulationsSystem`ja2`", "SimulationsSystem`ja3`"}]
Begin["`Private`"] (* Begin Private Context *) 
shadow=1;
var1=1;
pro1[]:=var1+var2+var3

End[] (* End Private Context *)

EndPackage[]

Package JA2

BeginPackage["SimulationsSystem`ja2`"]

shadow::usage="shadow"
var2::usage = "var2"
pro2::usage = "pro2"
(* Exported symbols added here with SymbolName::usage *)  
Needs["SimulationsSystem`ja1`"]
Needs["SimulationsSystem`ja3`"]  
EndPackage[]

BeginPackage["SimulationsSystem`ja2`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja3`"}]
Begin["`Private`"] (* Begin Private Context *) 
Needs["SimulationsSystem`ja3`"]
shadow=2;
var2=1;
pro2[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]

Package JA3

BeginPackage["SimulationsSystem`ja3`"]

shadow::usage="shadow"
var3::usage = "var3"
pro3::usage = "pro3"
(* Exported symbols added here with SymbolName::usage *)  
Needs["SimulationsSystem`ja1`"]
Needs["SimulationsSystem`ja2`"]

pro5::usage = "pro5  "  
EndPackage[]

BeginPackage["SimulationsSystem`ja3`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja2`"}]
Begin["`Private`"] (* Begin Private Context *) 
shadow=3;
var3=1;
pro3[]:=var1+var2+var3
pro5[]:=a
End[] (* End Private Context *)

EndPackage[]

Output:

In[1]:= << SimulationsSystem`ja1`
$ContextPath

During evaluation of In[1]:= shadow::shdw: Symbol shadow appears in multiple contexts {SimulationsSystem`ja2`,SimulationsSystem`ja1`}; definitions in context SimulationsSystem`ja2` may shadow or be shadowed by other definitions. >>

During evaluation of In[1]:= shadow::shdw: Symbol shadow appears in multiple contexts {SimulationsSystem`ja3`,SimulationsSystem`ja2`,SimulationsSystem`ja1`}; definitions in context SimulationsSystem`ja3` may shadow or be shadowed by other definitions. >>

Out[1]= {"SimulationsSystem`ja1`", "SimulationsSystem`ja2`", \
"SimulationsSystem`ja3`", "PacletManager`", "WebServices`", \
"System`", "Global`"}

In[2]:= pro1[]

Out[2]= 3

In[3]:= pro2[]

Out[3]= 3

In[4]:= pro3[]

Out[4]= 3

In[5]:= var1

Out[5]= 1

In[6]:= var2

Out[6]= 1

In[7]:= var3

Out[7]= 1

In[8]:= shadow

Out[8]= 1

It Works!

$\endgroup$
6
  • 1
    $\begingroup$ Really? Why do we need these tricks? I think the whole Package setup is not nice and completely oldfashioned and there should be something better. I wished these fundamental problems in software design would be addressed in some version of Mathematica. However, it has not changed much since 1987 or so. $\endgroup$ Sep 7, 2012 at 9:37
  • $\begingroup$ @RolfMertig I disagree with you this once, Rolf. I think the package system is very well designed and thought out, in that the user gets enormous control over all encapsulation mechanisms, most of which are exposed to us. The problem under discussion I view a consequence of this. Should we have a more rigid packaging system, it might have addressed this one, but it would restrict what one can possibly do. I do agree that such a system might be needed, but it can as well be implemented on top of the existing one (which, indeed, has not been done yet). $\endgroup$ Sep 7, 2012 at 9:43
  • $\begingroup$ @LeonidShifrin Sure, one can control everything, and I and you can (though I basically end up using full context names a lot), and one can manipulate $Path and alike. But try to explain this to someone new. And that "Workbench" does not help at all with subpackages. It is really really not enough for non-specialists. And even I get annoyed all the time when things get slightly more involved. There are subtle issues with SetDelayed and contexts, also circular Needs can be tricky. And by now I gave up with DeclarePackage. It is just too cumbersome to get it to do what one wants. $\endgroup$ Sep 7, 2012 at 10:36
  • $\begingroup$ @RolfMertig Yes, I agree, we are just given the raw system not really optimized for professional large-scale software development, and this is just one aspect of it. I hope this will change in the future. The good news is that everything we need can be implemented in the top-level Mathematica. $\endgroup$ Sep 7, 2012 at 11:28
  • 1
    $\begingroup$ @RolfMertig I would use Block and modify locally the relevant variables ($ContextPath, $Packages and $Context), plus modify globally some of them too ($ContextPath, $Packages). Can't offer a complete solution right now, will think about it, but it does not seem to be too hard. The key point is that the parsing (creation of symbols in the correct context) is completely controlled by the current values of $Context and $ContextPath, which both are exposed to the user. $\endgroup$ Sep 7, 2012 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.