# Saving data inside a notebook so that I don't have to run it again?

Piggybacking on this, I am somehow not fully convinced that I can't save data generated by a calculation in a mathematica file so that when I re-launch said file, I wouldn't have to run my calculations again.

I ask because I use NDSolve for 4th order non linear PDEs and sometimes I need to run it for really large times (in excess of a few hours, yes that sounds crazy but I am just trying to get results for a fluid dynamics problem here and I am not all that interested in being a computer scientist to reduce run times).

So after reading the linked article, I did this for an example problem:

sol = NDSolve[{D[u[t, x], t] == D[u[t, x], x, x], u[0, x] == 0,
u[t, 0] == Sin[t], u[t, 5] == 0}, u, {t, 0, 10}, {x, 0, 5}]

DumpSave["pde0.mx", sol]


Then I quit mathematica and relaunch it and load pde0.mx with Get

Get["pde0.mx"]


And when I plot,

Plot3D[Evaluate[u[t, x] /. sol], {t, 0, 10}, {x, 0, 5},
PlotRange -> All]


Voila, I get the plot as if I had run the simulation.

So.. did I run the simulation again by invoking sol through Get or was my kernel state saved?

-
Of course that solution (an InterpolatingFunction object) was saved in the file, so you did not have to solve for it again. My point in that answer was that DumpSave saves what you select to save, which is generally only a part of the kernel state. I also said there that often (perhaps in most cases) this can be sufficient. –  Leonid Shifrin May 23 '12 at 15:19
This old Mathgroup post, "Best way to save data in notebooks" by Yaroslav Bulatov might also be of interest, especially the reply of John Fultz –  TomD May 23 '12 at 18:59
@LeonidShifrin Yes, I am not disputing your answer, as a novice it was unclear to me! Thanks for your comment! –  drN May 23 '12 at 20:10
Thank you all for the multitude of answers! I am sure I'll take some time reading through all this! Thanks! –  drN May 23 '12 at 20:12

What was saved was the content of sol, which happens to contain the solution to your equation (you explicitly set it to that), and therefore is certainly sufficient for your plot.

Saving Kernel state however would involve saving things like the random seed, so the following would give the same output twice (using a hypothetical function SaveKernelState and corresponding LoadKernelState):

SaveKernelState["somefile"]
Print[RandomInteger[10, 10]]
(*
==> {5, 0, 1, 1, 7, 4, 4, 7, 9, 8}
*)
Print[RandomInteger[10, 10]]
(*
==> {5, 0, 1, 1, 7, 4, 4, 7, 9, 8}
*)


Also there are internal caches for things like FullSimplify, e.g.

FullSimplify[Sum[Sin[k^2 x],{k,0,8}]]//Timing
(*
{2.89218, Sin[x] + Sin[4 x] + Sin[9 x] + Sin[16 x] + Sin[25 x] +
Sin[36 x] + Sin[49 x] + Sin[64 x]}
*)
FullSimplify[Log[Sum[Sin[k^2 x],{k,0,8}]]]//Timing
(*
{0.028002, Log[Sin[x] + Sin[4 x] + Sin[9 x] + Sin[16 x] + Sin[25 x] +
Sin[36 x] + Sin[49 x] + Sin[64 x]]}
*)


Here, the result of the first FullSimplify was cached and reused in the second one. Saving full Kernel state would include saving those caches, so the second simplification would go much faster in the restarted session as well.

SetAttributes[PermanentSet,HoldAll];
PermanentSet[var_Symbol,value_]:=
(If[OwnValues[var] === {},
Module[{nb=EvaluationNotebook[]},
SelectionMove[nb, Before, EvaluationCell];
NotebookWrite[nb,Cell[ToString[Unevaluated[var]] <>
" = Uncompress[\"" <>
Compress[var=value] <>
"\"]",
"Input",
Editable->False]]]];
var)


This is used as follows: To set the (previously unassigned) variable a to the result of the time-consuming calculation Pause[2];1+1, just write

PermanentSet[a, Pause[2];1+1]


Executing this while a is not set will evaluate the expression and assign the result (2) to a, but will additionally add a cell before this one, containing

a = Uncompress["1:eJxTTMoPymRiYGAAAAtMAbA="]


(now in this case, a = 2 would have been shorter :-)). So when you evaluate the notebook in order, you'll first evaluate that line, setting a to 2, and only then the PermanentSet. Since PermanentSet now finds a already assigned, it doesn't evaluate the second argument again, but just returns the value.

Bugs and Limitations:

• If the evaluation contains side effects, those side effects will not be executed when the variable is set from the previous cell. Therefore this should only be used for side-effect-free calculations.
• This code depends on the previous cell being evaluated before this one. Otherwise the evaluation is restarted. However, with side-effect-free calculations, it should be safe to abort that calculation and execute the previous cell.
• This code doesn't work if the variable has a value, therefore if you want to replace an existing value, you have to unset the value first. However that unsetting must not happen in the same cell, but in a cell before. Otherwise it will undo the setting by the previous cell on re-evaluation.
• If the cell containing PermanentSet is an initialization cell, the generated cell should be an initialization cell as well. However it isn't (because I don't know how to do that).
• Also, this will place the selection immediately before the cell containing the call. Ideally it would save where the current selection is and restore it afterwards. However I don't know how to do that either.

Edit 2:

Based on Rojo's idea to store the data in notebook tagging rules (a feature which I didn't know about before), I've now written a different version of PermanentSet which resolves some of the problems with the previous one. It now saves both the expression and the resulting value in the tagging rules. This way, the function is evaluated again iff the expression has changed.

SetAttributes[PermanentSet,HoldAll];
PermanentSet[var_Symbol, value_] :=
Module[{nb=EvaluationNotebook[],
name=ToString[Unevaluated[var]],
expr=Compress[Unevaluated[value]]},
If[TrueQ[CurrentValue[nb, {TaggingRules, "Storage",
name <> "expression"}] == expr],
var = Uncompress@CurrentValue[nb, {TaggingRules, "Storage", name<>"value"}],
CurrentValue[nb, {TaggingRules, "Storage", name<>"expression"}] = expr;
CurrentValue[nb,{TaggingRules, "Storage", name<>"value"}] =
Compress[var=value]];
var];


Bugs and Limitations:

• As soon as the expression is evaluated, it won't be evaluated again until the expression is changed. That may be desired, but it also may be undesired (e.g. if re-evaluating because some variables changed). You can force a re-evaluation be PermanentSetting to a dummy value (e.g. Null) before re-evaluating. I don't see an easy way to automate that, though, because the variables may be used in functions called during the evaluation; therefore it's not possible to automatically determine which variables/values are needed. Maybe a TrackedVariables option like the one for DynamicModule would be useful for this.
• There probably should be a PermanentUnset counterpart to PermanentSet.
• As in the previous version, it is not a good idea to have side effects in the expression, because they will not happen again.
-
Since I've now incorporated so many ideas from others, I've made the post community wiki. –  celtschk May 23 '12 at 19:33

There is an easy way to keep your data in the notebook itself and NOT to save them in external file - using Compress. As @Leonid says here and I already mentioned this before in this answer for similar case with Interpolation function. Start from some output you need:

sol = NDSolve[{D[u[t, x], t] == D[u[t, x], x, x], u[0, x] == 0,
u[t, 0] == Sin[t], u[t, 5] == 0}, u, {t, 0, 10}, {x, 0, 5}];


and get compressed string of it

Compress[sol]


What ever output you get from it is a string - assign it to new variable say solCO:

And save that whole string containing cell in your notebook. Then when you open notebook use:

solUC = Uncompress[solCO];
Plot3D[Evaluate[u[t, x] /. solUC], {t, 0, 10}, {x, 0, 5}, PlotRange -> All]


Now you can make your string containing cell to an initialization cell via Menu > Cell > Cell Properties > Initialization cell so it will be executed as soon as you start working with that notebook.

-
How is this different from my answer, in its essence? –  Leonid Shifrin May 23 '12 at 16:52
I think it is the same but when I followed, your link I could not see in that code short Compress implementation. I wanted to give clear example here and also refer to a similar question I answered before. I'll add a note on that in the answer. I added a note on the answer body. I think all efforts are good ;-) –  Vitaliy Kaurov May 23 '12 at 17:20
Fair enough, I did not provide explicit examples in my answer. –  Leonid Shifrin May 23 '12 at 17:23

The option to save a variable, a value, in a notebook, that I find simple and deserves a chance is to store them in the notebook's tagging rules. You can compress it if you want, or you can autoload it through an initialization cell or through the NotebookDynamicExpression too. The core is this:

r = RandomReal[{-1, 1}, 1000000];
CurrentValue[InputNotebook[], {TaggingRules, "Storage", "r"}] = r;


The "Storage" extra tag is only to avoid unnecessary potential conflicts.

So, when you want to load it, you do

r=CurrentValue[InputNotebook[], {TaggingRules, "Storage", "r"}]


and you could also use it directly without loading it. Remember it's not a kernel variable until you load it.

If you want to use Compress to help, you just add it. For example,

 (* Helper, to avoid code repetition *)
SetAttributes[withMacros, HoldAllComplete];
withMacros[rules : {(_Rule | _RuleDelayed) ..}, code_] :=
Unevaluated[code] /. rules;

SetAttributes[{StoreInNotebook, GetFromNotebook, RestoreFromNotebook,
ClearNotebookStorage}, HoldFirst];
SetAttributes[{StoreInNotebook, RestoreFromNotebook,
ClearNotebookStorage}, Listable];

withMacros[{cv[s_] :>
CurrentValue[
EvaluationNotebook[], {TaggingRules, "Storage",
ToString@Unevaluated@s}]},

StoreInNotebook[s_Symbol] := cv[s] = Compress[s];

RestoreFromNotebook[s_Symbol] := (s = Uncompress@cv[s];);

GetFromNotebook[s_Symbol] := Uncompress@cv[s];

ClearNotebookStorage[] :=
CurrentValue[EvaluationNotebook[], {TaggingRules, "Storage"}] = {};

ClearNotebookStorage[s_Symbol] := cv[s] = {};
]


You could use this in this way:

x={1,2, 3};
StoreInNotebook[x];
x=.;
GetFromNotebook[x]


{1, 2, 3}

RestoreFromNotebook[x]
x


{1, 2, 3}

ClearNotebookStorage[x]

x1 = 8; x2 = 9;
StoreInNotebook[{x1, x2}];

ClearNotebookStorage[]


... This lacks proper messages as is. It is simple to change the code to add a per-variable flag that tells you if it has been saved compressed or uncompressed (imagine CurrentValue[nb, {TaggingRules, "Storage", "var", "CompressedFlag"}] and the value stored in CurrentValue[nb, {TaggingRules, "Storage", "var", "Value"}]. This way, StoreInNotebook could get an option to compress or not. Etc etc etc etc etc

-
Good stuff - +1. A stylistic comment: you could significantly reduce code duplication by generating part of that at definition - time with, e.g., replacement rules. –  Leonid Shifrin May 23 '12 at 18:18
@LeonidShifrin, I just edited. Any other tips? –  Rojo May 23 '12 at 18:31
Nothing else comes to mind. Interestingly, you did it exactly the way I had in mind, particularly I also would use the symbol cv, and make it depend on a parameter, not to explicitly break the scoping (even though the rules act earlier than bindings, so this does not matter here). Also, With[{cv:=...} (which breaks the scoping) won't likely work here without using SetDelayed@@Hold[...] for definitions, due to renamings performed by With. –  Leonid Shifrin May 23 '12 at 18:36
Thank you for sharing this method; I've now written a new version of PermanentSet which uses this method to store the values (see my answer). –  celtschk May 23 '12 at 19:30
This is so useful, but I have a problem with using it inside a function call: namely, if I use StoreInNotebook within a function, then try to call GetFromNotebook later on, I get the error Uncompress::string: String expected at position 1 in Uncompress[Inherited]. Any thoughts? –  Steve D Jan 27 '14 at 3:29

As an alternative to DumpSave, what I've done in the past was to Compress the relevant results and store them within the same notebook. You can optionally set things up so that your data/results would self-uncompress themselves upon being called the first time. For one example of such use, see this answer.

In any case, the advantage of this approach is that everything is stored within your notebook, so you don't have dependencies on external files (actually, one of the reasons I needed this set-up was to be able to communicate my results to my collaborators, and make sure that all they have to do is to run the code in the notebook. This worked quite well).

-
That answer never gave me the impression that it was saving the data within the notebook itself. Could you expand upon that? –  rcollyer May 23 '12 at 15:37
@rcollyer In that answer, it was not,indeed. To make self-uncompressing within a notebook to work, one would have to write a function which would create a cell with the relevant code, and place it into a proper place (e.g. make it initialization cell), and hook that function to the computations being performed. I did not myself use this scheme, but it should be doable. –  Leonid Shifrin May 23 '12 at 16:31
Well, after reading this, I've now written a function to do exactly that: SetAttributes[PermanentSet,HoldAll];PermanentSet[var_,value_]:=(If[a===Unevalua‌​ted[a],Module[{nb=EvaluationNotebook[]},SelectionMove[nb,Before,EvaluationCell];N‌​otebookWrite[nb,Cell[ToString[Unevaluated[var]]<>" = Uncompress[\""<>With[{v=var},Compress[var=value]]<>"\"]","Input",Editable->False‌​]]]];var). It assumes that the variable was previously undefined (if not, use =. on that variable in a previous cell). Also, the calculation should better not have any side effect. –  celtschk May 23 '12 at 17:51
@celtschk Yes, something along these lines. I wanted to write some more general functionality to do this sort of things, add some syntactic sugar etc, but don't have time for that right now. Your function may actually be sufficient for most situations. If you tested it and feel like including it into my answer, please do so. You probably meant var in place of a in your code, and a somewhat better test for whether or not a variable has a value can be found here –  Leonid Shifrin May 23 '12 at 18:12
@LeonidShifrin: I've now already added it to my answer. Thanks for catching the bug, I indeed meant var (I always used a as variable in my tests, so I didn't notice). I'll also look into the test you lined. –  celtschk May 23 '12 at 18:22