How to automatically do a computation, Quit[], restart and do a new computation

Question

Related questions have been asked before (eg Where does a package have to be loaded?, https://stackoverflow.com/questions/6152023/quit-kernel-in-mathematica-by-a-command-not-using-evaluation-menu, Restarting Mathematica automatically), but I don't think that any of the replies therein solves my question.

The Problem

I need to run a long computation over different datasets - on the order of 100. This involves calling a package, running a set of calculations for dataset n, saving the results, quitting the kernel, reloading the package, repeating the same calculation for dataset n+1, etc.

Quit[] guarantees that there is no contamination between calculations. Using Cleanslate (instead of Quit[]) is also a possibility.

My current solution

The way I've been dealing with it so far is to create an auxiliary file "control.txt" containing 1, then read that number 1 and assign it to controlN (iterator), rewrite control.txt with 1->2, read the package, run the computation for dataset 1 (call it foo - a function provided by Package`) and Quit[] as here:

controlN = ToExpression@Import["control.txt"];
Export["control.txt", controlN + 1];
Needs["Package`"];
foo[controlN];
Quit[]

then create 100 copies of this cell and evaluate them all. Since I have SetOptions[$FrontEnd, "ClearEvaluationQueueOnKernelQuit" -> False], this goes through all cells/datasets, restarting automatically after each Quit[].

While this does the job, it's messy, and surely there must be a simpler, neater way.

A more desirable solution

The obvious thing to do would be to use a loop, like say Do

Do[
Needs["Package`"];
foo[controlN];
Quit[],{controlN,100}]

but this doesn't work for 2 reasons. First, the kernel quits after controlN=1, so the cycle is interrupted. Cleanslate could be used instead.

But then even if one uses CleanSlate, there's another problem: by putting Needs and foo inside Do[...], foo is in context Global', not Package' as intended (' should be a backtick).

Calling it as Package'foo obviously changes the context, but it doesn't work, as the calculation depends on other functions from Package' which are not called explicitly here.

A related problem

I am aware of what's written here: http://reference.wolfram.com/language/tutorial/SettingUpWolframLanguagePackages.html. Their example is:

Begin["a`"];
Print[Context[x]];
End[];

which returns a'. But by placing it inside Do[]

Do[Begin["a`"];
Print[Context[x]];
End[], {m, 1}]

it returns Global'. Can one force Begin (or equivalently the package) to be evaluated before the remaining lines inside the Do loop?

Is there a way to solve the initial problem which is close to the "more desirable solution" mentioned above?

Answer 1

You can start a fresh extra kernel programmatically from your current notebook using MathLink. You can then load packages in that kernel, perform necessary calculations, get the result back to your main notebook, and close the extra kernel. Here is a function which automates this process (based on freshKernelEvaluate from this answer by @jkuczm)

ClearAll[newKernelEvaluate];
Attributes[newKernelEvaluate] = HoldAll;
newKernelEvaluate[context_String, expr_] := Module[
    {link, result},
    link = LinkLaunch[First@$CommandLine <> " -mathlink -noprompt"];
    LinkWrite[link, Unevaluated@EvaluatePacket@Needs[context]];
    LinkRead@link;
    LinkWrite[link, Unevaluated@EvaluatePacket@expr];
    result = LinkRead@link;
    LinkClose@link;
    Replace[result, ReturnPacket@x_ :> x]
];

Note that Needs[] command is sent as a separate packet in order to avoid shadowing problems.

You can now use newKernelEvaluate to perform calculations in fresh kernels. In the following example "ComputerArithmetic`" package is loaded and ComputerArithmetic`Ulp function is calculated. Note that the context of Ulp is recognized automatically as you want.

Table[
    With[{i=controlN},
        newKernelEvaluate["ComputerArithmetic`",
            Ulp[i 1000.]
            ]
        ],
    {controlN,1,5}
    ]

{1.13687*10^-13, 2.27374*10^-13, 4.54747*10^-13, 4.54747*10^-13, 9.09495*10^-13}