59
$\begingroup$

Piggybacking on this, I am somehow not fully convinced that I can't save data generated by a calculation in a mathematica file so that when I re-launch said file, I wouldn't have to run my calculations again.

I ask because I use NDSolve for 4th order non linear PDEs and sometimes I need to run it for really large times (in excess of a few hours, yes that sounds crazy but I am just trying to get results for a fluid dynamics problem here and I am not all that interested in being a computer scientist to reduce run times).

So after reading the linked article, I did this for an example problem:

sol = NDSolve[{D[u[t, x], t] == D[u[t, x], x, x], u[0, x] == 0, 
   u[t, 0] == Sin[t], u[t, 5] == 0}, u, {t, 0, 10}, {x, 0, 5}]

DumpSave["pde0.mx", sol]

Then I quit mathematica and relaunch it and load pde0.mx with Get

Get["pde0.mx"]

And when I plot,

Plot3D[Evaluate[u[t, x] /. sol], {t, 0, 10}, {x, 0, 5}, 
 PlotRange -> All]

Voila, I get the plot as if I had run the simulation.

So.. did I run the simulation again by invoking sol through Get or was my kernel state saved?

$\endgroup$
  • 2
    $\begingroup$ Of course that solution (an InterpolatingFunction object) was saved in the file, so you did not have to solve for it again. My point in that answer was that DumpSave saves what you select to save, which is generally only a part of the kernel state. I also said there that often (perhaps in most cases) this can be sufficient. $\endgroup$ – Leonid Shifrin May 23 '12 at 15:19
  • 1
    $\begingroup$ This old Mathgroup post, "Best way to save data in notebooks" by Yaroslav Bulatov might also be of interest, especially the reply of John Fultz $\endgroup$ – user1066 May 23 '12 at 18:59
  • $\begingroup$ @LeonidShifrin Yes, I am not disputing your answer, as a novice it was unclear to me! Thanks for your comment! $\endgroup$ – dearN May 23 '12 at 20:10
  • $\begingroup$ Thank you all for the multitude of answers! I am sure I'll take some time reading through all this! Thanks! $\endgroup$ – dearN May 23 '12 at 20:12
  • $\begingroup$ Related: "Storing data in the Notebook in readable form." $\endgroup$ – Alexey Popkov Feb 29 '16 at 7:44
29
$\begingroup$

What was saved was the content of sol, which happens to contain the solution to your equation (you explicitly set it to that), and therefore is certainly sufficient for your plot.

Saving Kernel state however would involve saving things like the random seed, so the following would give the same output twice (using a hypothetical function SaveKernelState and corresponding LoadKernelState):

SaveKernelState["somefile"]
Print[RandomInteger[10, 10]]
(*
==> {5, 0, 1, 1, 7, 4, 4, 7, 9, 8}
*)
LoadKernelState["somefile"]
Print[RandomInteger[10, 10]]
(*
==> {5, 0, 1, 1, 7, 4, 4, 7, 9, 8}
*)

Also there are internal caches for things like FullSimplify, e.g.

FullSimplify[Sum[Sin[k^2 x],{k,0,8}]]//Timing       
(*
{2.89218, Sin[x] + Sin[4 x] + Sin[9 x] + Sin[16 x] + Sin[25 x] + 
Sin[36 x] + Sin[49 x] + Sin[64 x]}
*)
FullSimplify[Log[Sum[Sin[k^2 x],{k,0,8}]]]//Timing 
(*
{0.028002, Log[Sin[x] + Sin[4 x] + Sin[9 x] + Sin[16 x] + Sin[25 x] + 
Sin[36 x] + Sin[49 x] + Sin[64 x]]}
*)

Here, the result of the first FullSimplify was cached and reused in the second one. Saving full Kernel state would include saving those caches, so the second simplification would go much faster in the restarted session as well.

Edit: After reading Leonid's answer and the comments (as well as the answer he linked), I've now written a function which does exactly what you ask for in your title: Save the data inside your notebook:

SetAttributes[PermanentSet,HoldAll];
PermanentSet[var_Symbol,value_]:=
  (If[OwnValues[var] === {},
      Module[{nb=EvaluationNotebook[]},
             SelectionMove[nb, Before, EvaluationCell];
             NotebookWrite[nb,Cell[ToString[Unevaluated[var]] <>
                                   " = Uncompress[\"" <>
                                   Compress[var=value] <>
                                   "\"]",
                                   "Input",
                                   Editable->False]]]];
   var)

This is used as follows: To set the (previously unassigned) variable a to the result of the time-consuming calculation Pause[2];1+1, just write

PermanentSet[a, Pause[2];1+1]

Executing this while a is not set will evaluate the expression and assign the result (2) to a, but will additionally add a cell before this one, containing

a = Uncompress["1:eJxTTMoPymRiYGAAAAtMAbA="]

(now in this case, a = 2 would have been shorter :-)). So when you evaluate the notebook in order, you'll first evaluate that line, setting a to 2, and only then the PermanentSet. Since PermanentSet now finds a already assigned, it doesn't evaluate the second argument again, but just returns the value.

Bugs and Limitations:

  • If the evaluation contains side effects, those side effects will not be executed when the variable is set from the previous cell. Therefore this should only be used for side-effect-free calculations.
  • This code depends on the previous cell being evaluated before this one. Otherwise the evaluation is restarted. However, with side-effect-free calculations, it should be safe to abort that calculation and execute the previous cell.
  • This code doesn't work if the variable has a value, therefore if you want to replace an existing value, you have to unset the value first. However that unsetting must not happen in the same cell, but in a cell before. Otherwise it will undo the setting by the previous cell on re-evaluation.
  • If the cell containing PermanentSet is an initialization cell, the generated cell should be an initialization cell as well. However it isn't (because I don't know how to do that).
  • Also, this will place the selection immediately before the cell containing the call. Ideally it would save where the current selection is and restore it afterwards. However I don't know how to do that either.

Edit 2:

Based on Rojo's idea to store the data in notebook tagging rules (a feature which I didn't know about before), I've now written a different version of PermanentSet which resolves some of the problems with the previous one. It now saves both the expression and the resulting value in the tagging rules. This way, the function is evaluated again iff the expression has changed.

SetAttributes[PermanentSet,HoldAll];
PermanentSet[var_Symbol, value_] :=
  Module[{nb=EvaluationNotebook[],
          name=ToString[Unevaluated[var]],
          expr=Compress[Unevaluated[value]]},
    If[TrueQ[CurrentValue[nb, {TaggingRules, "Storage",
                               name <> "expression"}] == expr],
      var = Uncompress@CurrentValue[nb, {TaggingRules, "Storage", name<>"value"}],
      CurrentValue[nb, {TaggingRules, "Storage", name<>"expression"}] = expr;
      CurrentValue[nb,{TaggingRules, "Storage", name<>"value"}] =
        Compress[var=value]];
    var];

Bugs and Limitations:

  • As soon as the expression is evaluated, it won't be evaluated again until the expression is changed. That may be desired, but it also may be undesired (e.g. if re-evaluating because some variables changed). You can force a re-evaluation be PermanentSetting to a dummy value (e.g. Null) before re-evaluating. I don't see an easy way to automate that, though, because the variables may be used in functions called during the evaluation; therefore it's not possible to automatically determine which variables/values are needed. Maybe a TrackedVariables option like the one for DynamicModule would be useful for this.
  • There probably should be a PermanentUnset counterpart to PermanentSet.
  • As in the previous version, it is not a good idea to have side effects in the expression, because they will not happen again.
$\endgroup$
  • $\begingroup$ Since I've now incorporated so many ideas from others, I've made the post community wiki. $\endgroup$ – celtschk May 23 '12 at 19:33
29
$\begingroup$

There is an easy way to keep your data in the notebook itself and NOT to save them in external file - using Compress. As @Leonid says here and I already mentioned this before in this answer for similar case with Interpolation function. Start from some output you need:

sol = NDSolve[{D[u[t, x], t] == D[u[t, x], x, x], u[0, x] == 0, 
   u[t, 0] == Sin[t], u[t, 5] == 0}, u, {t, 0, 10}, {x, 0, 5}];

and get compressed string of it

Compress[sol]

What ever output you get from it is a string - assign it to new variable say solCO:

enter image description here

And save that whole string containing cell in your notebook. Then when you open notebook use:

solUC = Uncompress[solCO];
Plot3D[Evaluate[u[t, x] /. solUC], {t, 0, 10}, {x, 0, 5}, PlotRange -> All]

enter image description here

Now you can make your string containing cell to an initialization cell via Menu > Cell > Cell Properties > Initialization cell so it will be executed as soon as you start working with that notebook.


As an alternative, you can also store the compressed data inside a Button object, using code of the form

With[{data=Compress[sol]},
  Button["Restore"], Set[sol, Uncompress[data]]
]

which produces a simple button of the form

Standard Mathematica button reading 'Restore'

that takes little room on the notebook (and which, in particular, does not force the front end to display large cells of gibberish which might slow it down), but whose (non-displayed) FullForm contains all the information of the Compressed data on sol. Clicking on the button uncompresses the data and assigns the value to that variable.

This gets a little bit trickier if the data has been stored memoized into the case-by-case definitions of a function, but this can also be done by turning the function's FullDefinition into a string:

With[{data = Compress[ToString[InputForm[FullDefinition[function]]]]},
  Button["Restore", ToExpression[Uncompress[data]]]
]
$\endgroup$
  • $\begingroup$ How is this different from my answer, in its essence? $\endgroup$ – Leonid Shifrin May 23 '12 at 16:52
  • 1
    $\begingroup$ I think it is the same but when I followed, your link I could not see in that code short Compress implementation. I wanted to give clear example here and also refer to a similar question I answered before. I'll add a note on that in the answer. I added a note on the answer body. I think all efforts are good ;-) $\endgroup$ – Vitaliy Kaurov May 23 '12 at 17:20
  • $\begingroup$ Fair enough, I did not provide explicit examples in my answer. $\endgroup$ – Leonid Shifrin May 23 '12 at 17:23
  • 1
    $\begingroup$ As a simple alternative to keeping the data in a (long) cell which can slow down the front end, you can put it in a button: With[{data=Compress[sol], Button["Restore"], Set[sol, Uncompress[data]]. The compressed data is then part of the undisplayed FullForm of the Button object, and getting the data back is a single click away. $\endgroup$ – Emilio Pisanty May 3 '16 at 21:27
  • $\begingroup$ @EmilioPisanty nice idea! On another note, have you thought of anouncing your packages at our Add Ons Group ? $\endgroup$ – Vitaliy Kaurov May 3 '16 at 22:55
19
$\begingroup$

The option to save a variable, a value, in a notebook, that I find simple and deserves a chance is to store them in the notebook's tagging rules. You can compress it if you want, or you can autoload it through an initialization cell or through the NotebookDynamicExpression too. The core is this:

r = RandomReal[{-1, 1}, 1000000];
CurrentValue[InputNotebook[], {TaggingRules, "Storage", "r"}] = r;

The "Storage" extra tag is only to avoid unnecessary potential conflicts.

So, when you want to load it, you do

r=CurrentValue[InputNotebook[], {TaggingRules, "Storage", "r"}]

and you could also use it directly without loading it. Remember it's not a kernel variable until you load it.

If you want to use Compress to help, you just add it. For example,

 (* Helper, to avoid code repetition *)
SetAttributes[withMacros, HoldAllComplete];
withMacros[rules : {(_Rule | _RuleDelayed) ..}, code_] :=
  Unevaluated[code] /. rules;

SetAttributes[{StoreInNotebook, GetFromNotebook, RestoreFromNotebook, 
   ClearNotebookStorage}, HoldFirst];
SetAttributes[{StoreInNotebook, RestoreFromNotebook, 
   ClearNotebookStorage}, Listable];

withMacros[{cv[s_] :> 
   CurrentValue[
    EvaluationNotebook[], {TaggingRules, "Storage", 
     ToString@Unevaluated@s}]},

 StoreInNotebook[s_Symbol] := cv[s] = Compress[s];

 RestoreFromNotebook[s_Symbol] := (s = Uncompress@cv[s];);

 GetFromNotebook[s_Symbol] := Uncompress@cv[s];

 ClearNotebookStorage[] := 
  CurrentValue[EvaluationNotebook[], {TaggingRules, "Storage"}] = {};

 ClearNotebookStorage[s_Symbol] := cv[s] = {};
 ]

You could use this in this way:

x={1,2, 3};
StoreInNotebook[x];
x=.;
GetFromNotebook[x]

{1, 2, 3}

RestoreFromNotebook[x]
x

{1, 2, 3}

ClearNotebookStorage[x]

x1 = 8; x2 = 9;
StoreInNotebook[{x1, x2}];

ClearNotebookStorage[]

... This lacks proper messages as is. It is simple to change the code to add a per-variable flag that tells you if it has been saved compressed or uncompressed (imagine CurrentValue[nb, {TaggingRules, "Storage", "var", "CompressedFlag"}] and the value stored in CurrentValue[nb, {TaggingRules, "Storage", "var", "Value"}]. This way, StoreInNotebook could get an option to compress or not. Etc etc etc etc etc

$\endgroup$
  • $\begingroup$ Good stuff - +1. A stylistic comment: you could significantly reduce code duplication by generating part of that at definition - time with, e.g., replacement rules. $\endgroup$ – Leonid Shifrin May 23 '12 at 18:18
  • $\begingroup$ @LeonidShifrin, I just edited. Any other tips? $\endgroup$ – Rojo May 23 '12 at 18:31
  • $\begingroup$ Nothing else comes to mind. Interestingly, you did it exactly the way I had in mind, particularly I also would use the symbol cv, and make it depend on a parameter, not to explicitly break the scoping (even though the rules act earlier than bindings, so this does not matter here). Also, With[{cv:=...} (which breaks the scoping) won't likely work here without using SetDelayed@@Hold[...] for definitions, due to renamings performed by With. $\endgroup$ – Leonid Shifrin May 23 '12 at 18:36
  • $\begingroup$ Thank you for sharing this method; I've now written a new version of PermanentSet which uses this method to store the values (see my answer). $\endgroup$ – celtschk May 23 '12 at 19:30
  • $\begingroup$ This is so useful, but I have a problem with using it inside a function call: namely, if I use StoreInNotebook within a function, then try to call GetFromNotebook later on, I get the error Uncompress::string: String expected at position 1 in Uncompress[Inherited]. Any thoughts? $\endgroup$ – Steve D Jan 27 '14 at 3:29
13
$\begingroup$

As an alternative to DumpSave, what I've done in the past was to Compress the relevant results and store them within the same notebook. You can optionally set things up so that your data/results would self-uncompress themselves upon being called the first time. For one example of such use, see this answer.

In any case, the advantage of this approach is that everything is stored within your notebook, so you don't have dependencies on external files (actually, one of the reasons I needed this set-up was to be able to communicate my results to my collaborators, and make sure that all they have to do is to run the code in the notebook. This worked quite well).

$\endgroup$
  • $\begingroup$ That answer never gave me the impression that it was saving the data within the notebook itself. Could you expand upon that? $\endgroup$ – rcollyer May 23 '12 at 15:37
  • $\begingroup$ @rcollyer In that answer, it was not,indeed. To make self-uncompressing within a notebook to work, one would have to write a function which would create a cell with the relevant code, and place it into a proper place (e.g. make it initialization cell), and hook that function to the computations being performed. I did not myself use this scheme, but it should be doable. $\endgroup$ – Leonid Shifrin May 23 '12 at 16:31
  • $\begingroup$ Well, after reading this, I've now written a function to do exactly that: SetAttributes[PermanentSet,HoldAll];PermanentSet[var_,value_]:=(If[a===Unevaluated[a],Module[{nb=EvaluationNotebook[]},SelectionMove[nb,Before,EvaluationCell];NotebookWrite[nb,Cell[ToString[Unevaluated[var]]<>" = Uncompress[\""<>With[{v=var},Compress[var=value]]<>"\"]","Input",Editable->False]]]];var). It assumes that the variable was previously undefined (if not, use =. on that variable in a previous cell). Also, the calculation should better not have any side effect. $\endgroup$ – celtschk May 23 '12 at 17:51
  • $\begingroup$ @celtschk Yes, something along these lines. I wanted to write some more general functionality to do this sort of things, add some syntactic sugar etc, but don't have time for that right now. Your function may actually be sufficient for most situations. If you tested it and feel like including it into my answer, please do so. You probably meant var in place of a in your code, and a somewhat better test for whether or not a variable has a value can be found here $\endgroup$ – Leonid Shifrin May 23 '12 at 18:12
  • $\begingroup$ @LeonidShifrin: I've now already added it to my answer. Thanks for catching the bug, I indeed meant var (I always used a as variable in my tests, so I didn't notice). I'll also look into the test you lined. $\endgroup$ – celtschk May 23 '12 at 18:22
8
$\begingroup$

Update: I maintain the latest version of this function on my blog.


SaveToCell is a convenience-minded implementation of Vitaliy's idea. The code is at the end.

Description

SaveToCell[var] creates an input cell that re-assigns the value of var. The right-hand-side of the assignment is displayed concisely in the notebook.

Usage example

range = Range[1000];

SaveToCell[range]

This creates a new input cell:

Mathematica graphics

Evaluating it re-assigns the value of range. The small panel with the label "data" can also be copied and pasted somewhere else in the notebook. It contains all the data in compressed form.

We can set a custom label instead of the default "data", using the second argument:

SaveToCell[range, Short[range]]

Mathematica graphics

Hovering the panel shows the time when it was saved.

enter image description here

Any options will be passed down to the generated cell, so if you are worried about accidentally deleting your data, you can use

SaveToCell[var, Deletable -> False]

(To delete such a cell, select its bracket, press Ctrl-Shift-E or Command-Shift-E on Mac to show its expression, and remove the Deletable option.)

Possible issues

SaveToCell re-evaluates the contents of the variable. Thus the following

var = {1, a};    
a = 5;    
SaveToCell[var]

will save {1,5} instead of {1,a} where a is a symbol.

In fact, the little panel is equivalent not even to {1,5} but to Uncompress["1:eJxTTMoPSmNiYGAoZgESPpnFJZmMQEYmK5AAAE4GBJg="]. This means that it will behave differently if copied and pasted inside of a Hold[...].

Code

ClearAll[SaveToCell]

SaveToCell::usage =
    "SaveToCell[variable] creates an input cell that reassigns the current value of variable.\n" <>
    "SaveToCell[variables, display] shows 'display' on the right-hand-side of the assignment.";

SetAttributes[SaveToCell, HoldFirst]
SaveToCell[var_, name : Except[_?OptionQ] : "data", opt : OptionsPattern[]] :=
    With[{data = Compress[var],
      panel = ToBoxes@Tooltip[Panel[name, FrameMargins -> Small], DateString[]]},
      CellPrint@Cell[
        BoxData@RowBox[{
          MakeBoxes[var],
          "=",
          InterpretationBox[panel, Uncompress[data]],
          ";"
        }],
        "Input",
        GeneratedCell -> False
        opt,
        CellLabel -> "(saved)", CellLabelAutoDelete -> False
      ]
    ]
$\endgroup$
8
$\begingroup$

In Mathematica 11.3 or later, use Iconize. It creates a short display form of a large expression that can be copied to elsewhere within the notebook, can be saved with the notebook, and can be used in the same way as the fully displayed version of that expression.

enter image description here

Typically one would save a notebook with a cell that assigns an iconized version of a large expression to a variable name.

enter image description here

Iconize is very similar to the SaveToCell functionality I described in my other answer.

$\endgroup$
  • $\begingroup$ I have been linked here since I had the same question as OP, and while I find Iconize the by far most convenient way to achieve what I need, it however does have the downside that saving and opening the notebook can take quite a bit of time, if I iconized big data sets coming from NDSolve. Is there any way to work around this downside, or would I have to use a different method than Iconize then? $\endgroup$ – Britzel Jun 6 at 13:14
  • $\begingroup$ @Britzel I think that there is no way around the slow notebook saving/loading when the notebook contains a lot of data ... your only option is to save to a different file that is stored next to the notebook. $\endgroup$ – Szabolcs Jun 6 at 15:34
  • $\begingroup$ Understand, thanks for the tip! $\endgroup$ – Britzel Jun 6 at 15:43
4
$\begingroup$

In Mathematica 11.2 or later can use PersistentValue to persist a value stored.

enter image description here

It is very convenient to use, and it can be directly assigned.

enter image description here

The data exists in the form of Hold, not in compressed form, so the documentation may be much larger than some other methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.