How to automatically do a computation, Quit[], restart and do a new computation

Related questions have been asked before (eg Where does a package have to be loaded?, https://stackoverflow.com/questions/6152023/quit-kernel-in-mathematica-by-a-command-not-using-evaluation-menu, Restarting Mathematica automatically), but I don't think that any of the replies therein solves my question.

The Problem

I need to run a long computation over different datasets - on the order of 100. This involves calling a package, running a set of calculations for dataset n, saving the results, quitting the kernel, reloading the package, repeating the same calculation for dataset n+1, etc.

Quit[] guarantees that there is no contamination between calculations. Using Cleanslate (instead of Quit[]) is also a possibility.

My current solution

The way I've been dealing with it so far is to create an auxiliary file "control.txt" containing 1, then read that number 1 and assign it to controlN (iterator), rewrite control.txt with 1->2, read the package, run the computation for dataset 1 (call it foo - a function provided by Package) and Quit[] as here:

controlN = ToExpression@Import["control.txt"];
Export["control.txt", controlN + 1];
Needs["Package"];
foo[controlN];
Quit[]


then create 100 copies of this cell and evaluate them all. Since I have SetOptions[$FrontEnd, "ClearEvaluationQueueOnKernelQuit" -> False], this goes through all cells/datasets, restarting automatically after each Quit[]. While this does the job, it's messy, and surely there must be a simpler, neater way. A more desirable solution The obvious thing to do would be to use a loop, like say Do Do[ Needs["Package"]; foo[controlN]; Quit[],{controlN,100}]  but this doesn't work for 2 reasons. First, the kernel quits after controlN=1, so the cycle is interrupted. Cleanslate could be used instead. But then even if one uses CleanSlate, there's another problem: by putting Needs and foo inside Do[...], foo is in context Global', not Package' as intended (' should be a backtick). Calling it as Package'foo obviously changes the context, but it doesn't work, as the calculation depends on other functions from Package' which are not called explicitly here. A related problem I am aware of what's written here: http://reference.wolfram.com/language/tutorial/SettingUpWolframLanguagePackages.html. Their example is: Begin["a"]; Print[Context[x]]; End[];  which returns a'. But by placing it inside Do[] Do[Begin["a"]; Print[Context[x]]; End[], {m, 1}]  it returns Global'. Can one force Begin (or equivalently the package) to be evaluated before the remaining lines inside the Do loop? Is there a way to solve the initial problem which is close to the "more desirable solution" mentioned above? • I would consider running Mathematica in batch mode from the command line. – yohbs May 12 '17 at 3:12 • You can start a new fresh kernel programmatically from your current notebook using MathLink. For example see freshKernelEvaluate function from this answer by @jkuczm – Shadowray May 12 '17 at 6:39 • – Alexey Popkov May 12 '17 at 9:14 • @yohbs - I also considered that. It works and solves the problem, but I was looking for a way to do everything in an elegant and self-contained manner. – Gaius May 12 '17 at 13:07 • @AlexeyPopkov - yes, I mentioned in the beginning that there were related questions. I had read yours, and your original question was very similar to mine, but I don't think that any of the answers provided a solution close to the "desirable solution" mentioned above. Don't you agree? – Gaius May 12 '17 at 13:13 1 Answer You can start a fresh extra kernel programmatically from your current notebook using MathLink. You can then load packages in that kernel, perform necessary calculations, get the result back to your main notebook, and close the extra kernel. Here is a function which automates this process (based on freshKernelEvaluate from this answer by @jkuczm) ClearAll[newKernelEvaluate]; Attributes[newKernelEvaluate] = HoldAll; newKernelEvaluate[context_String, expr_] := Module[ {link, result}, link = LinkLaunch[First@$CommandLine <> " -mathlink -noprompt"];
Replace[result, ReturnPacket@x_ :> x]
];


Note that Needs[] command is sent as a separate packet in order to avoid shadowing problems.

You can now use newKernelEvaluate to perform calculations in fresh kernels. In the following example "ComputerArithmetic" package is loaded and ComputerArithmeticUlp function is calculated. Note that the context of Ulp is recognized automatically as you want.

Table[
With[{i=controlN},
newKernelEvaluate["ComputerArithmetic",
Ulp[i 1000.]
]
],
{controlN,1,5}
]


{1.13687*10^-13, 2.27374*10^-13, 4.54747*10^-13, 4.54747*10^-13, 9.09495*10^-13}

• This old answer of mine contains elaborated implementation of something like newKernelEvaluate but with proper handling of Messages and Prints and also with proper passing down and handling of Aborts. – Alexey Popkov May 13 '17 at 11:44
• I voted for your solution. I'd also like to add an analogous solution derived from question 82803. Look at the answer by @kuba, and his function quitAndEvaluate. By changing quitAndEvaluate[after___] to something like quitAndEvaluate[context_, after___] and adding MakeBoxes[context;] before MakeBoxes[after;...] creates a separate cell to load context before after___. continued below – Gaius May 14 '17 at 13:49
• continuation More arguments can be added to quitAndEvaluate in this way if necessary. Then the original question can be solved with Do[ insert quitAndEvaluate definition here ; quitAndEvaluate[Needs["Package'"], controlN, foo[controlN]] ,{controlN,100}]. Where I added a second argument to pass the value of controlN. – Gaius May 14 '17 at 14:05
• Is there a way to make this work with .m files which are not packages? I have a lot of .m files containing definitions which are not proper packages (yes I guess that is the mistake). I would like to do something like newKernelEvaluate[ Get[filename.m]; f[expr]] where f is defined in filename.m. Would this be a simple adaption or is it essential that the packages load into a different scope than Global`? – Kvothe Jun 18 at 10:51