How does Needs["foo`"] find the file that defines context "foo`"?

Question

The post's title says it all, but if more detail is needed:

Suppose I have the following in some file /path/to/foo.m:

Begin["foo`"];
hello = Function[{}, Print["Hello world!"]];
End[];

...and in some other file, say, ./client.m, I put the following

Needs["foo`"];
foo`hello[];

Then, when the Needs expression gets evaluated, I get the error

Needs::nocont: Context foo` was not created when Needs was evaluated.

I'm too surprised by this error, since I see no good way for Mathematica to find where foo` is defined.

(BTW, putting /path/to in $Path does not change the results described above.)

What else must I put in ./client.m, besides Needs["foo`"], so that the expression foo`hello[] evaluates properly?

---

EDIT: I should explain that the reason I'm using Begin instead of BeginPackage is that I want to enforce full qualified names (e.g. foo`hello, rather than plain hello) as the only way to refer to imported functions.

Answer 1

This post answers specifically the title question:

How does Needs["foo`"] find the file that defines context "foo`"?

---

Get, Needs, Install, OpenRead, etc. all use FindFile. How FindFile resolves file names is discussed in:

• FindFile, Details section

• Naming and Finding Files

• Files for Packages

---

I don't know the full details (it's complicated), but roughly FindFile translates a context to a file path as follows:

FindFile["foo`"] either

• Looks on $Path for foo.m, foo.wl, foo.mx or directory foo
• If a paclet declares the context foo` in its Kernel extension, it translates to the corresponding Root directory: See PacletInfo.m documentation project, Kernel extension section.

Then if the result was a directory dir, it continues to look for

• dir/init.m (or init.wl)
• dir/Kernel/init.m (or init.wl)

If the result was a directory dir.mx, it continues to look for

• dir.mx/$SystemID/dir.mx (insert the value of $SystemID)

This is useful because .mx files are not compatible across different platforms.

---

FindFile["foo"], where foo is not a context,

• Looks on $Path for foo.

If the result is a directory, then it continues to look for

• foo/$SystemID/foo

This is useful with Install, when we need a separate executable for each $SystemID, but we want to be able to use the same name to refer to them on any platform.

---

Notes

The context of a package, i.e. the context given in BeginPackage that will contain the public package symbols, does not play any role in how FindFile resolves a context name to a file path.

However, Needs is different from Get in that it expects the context passed to it to appear in $Packages after the package has been loaded. This is why one needs to use BeginPackage and not merely Begin when writing a package. BeginPackage will permanently add that context both to $ContextPath and $Packages. Needs uses $Packages to determine if a package has already been loaded and avoid double-loading.

In short, the file names and the context of a package don't strictly need to be the same. But if they aren't, Mathematica will sometimes get confused.

The rules described above explain why the standard application directory structure is as described here. There is nothing strictly enforcing this particular structure, but the system is designed in a way that it expects to find this structure in any package.

---

Some interesting undocumented variables:

• Internal`$PackageDependencies is updated by BeginPackage based on its second argument

• $LoadedFiles is updated by Get. To get a more useful list, use Select[$LoadedFiles, Not@StringStartsQ[#, $InstallationDirectory] &]

Answer 2

The problem is not that Mathematica can not find the file, which it would indicate with another message. The problem is that Needs insists in your package to create the context it loads and put it onto $Packages.

To achieve that you need BeginPackage["foo`"] and the corresponding EndPackage in that file. Using that standard idiom will also make it possible to use the function name without the full context. That will of course also become a more minimal solution for the OPs problem.

From your comment I understand that you want to load the package but not have the package context in $ContextPath. The answer of Simoon Rochester shows you several ways to achieve that. Here are some more:

use Needs as usual, but remove the package name from $ContextPath afterwards:

Needs["foo`"]; $ContextPath = DeleteCases[$ContextPath,"foo`"]

the following will do the same and is somewhat more elegant and in some senses more reliable:

Block[{$ContextPath=$ContextPath},Needs["foo`"]]

but using Block in that way might have additional subtleties in specific situations.

Another thing you can do is to use two contexts, one for loading, the other one as the namespace of your symbols. If you put the following into file loadfoo.m:

BeginPackage["loadfoo`"];
EndPackage[];
Begin["foo`"];
hello = Function[{}, Print["Hello world!"]];
End[];

then

Needs["loadfoo`"]

will load with no error or warning messages and put "loadfoo`" onto $ContextPath but with no symbols in it. That file will also define foo`hello which you can use as you seem to intend.

EDIT jkuczm has silently changed a note given above which originally read that Needs expect the context to be added to $ContextPath to the more correct statement that it wants the context to be added to $Packages. Actually that is all that Needs really checks, a fact that I was not aware of as the pair of BeginPackage and EndPackage do more than that and the message that Needs just states that it wants the context to be created. Acutally you don't need to create any symbols in that context nor put it onto $ContextPath as I always assumed. So a file "test.m" anywhere in $Path will be loaded with no error messages by Needs if it contains just this:

Unprotect[$Packages];
AppendTo[$Packages,"test`"];
Protect[$Packages];

Using that idea, putting the following into a file "foo.m" should also solve the OPs problem without any special care by the user who loads and without adding an empty context to $ContextPath:

Unprotect[$Packages];
AppendTo[$Packages,"foo`"];
Protect[$Packages];
Begin["foo`"];
hello = Function[{}, Print["Hello world!"]];
End[];

which has the advantage that it can be loaded with just Needs["foo`"] and avoids any unnecessary side effects.

Answer 3

You've got more than one question here, but I'm not sure which problem you're really trying to solve.

• From the title: "How does Needs["foo`"] find the file that defines context foo`?"

Needs looks for a file in the $Path or in the current working directory named foo.m or foo.wl. (As a complication, there is also a list of Paclet directories that it looks in, but you can ignore that if you're not using PacletInfo.m files. The paclet files can be used to register packages where the file names don't correspond to the contexts, however, which may address one of your other questions.)

• What else must I put in ./client.m, besides Needs["foo`"], so that the expression foo`hello[] evaluates properly?

You don't need to put anything else. foo`hello[] should already be evaluating properly with what you have (assuming foo.m has been found on the $Path). The warning message that you get when you evaluate Needs doesn't affect that.

Likewise for the question from your comment about making the the use of full context manditory -- you have already done that, by using Begin instead of BeginPackage, so that the context foo` is not added to the $Contexts variable.

If it is the warning message that is primarily concerning you, the best approach is to use Get["foo`"] rather than Needs, since Needs is set up to expect the context to be registered, while Get doesn't care.

If you want to use Needs because you only want to load foo.m once in the session, you could instead use the Once function (new in version 11): Once[Get["foo`"]]. Or you could just use Quiet[Needs["foo`"]] or Off[Needs::nocont] to suppress the warning message.