Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Mathematica has numerous built-in data for science and math. Every first call to a data function in a new Mathematica session will "re-initialize indices" (download the data). Is there a way to store this data locally to save time on the next data call after Mathematica is restarted?

enter image description here

======== Edit: further thoughts with Leonid's answer ========

After Leonid posted his answer, I decided to update this question – the title and content. I would ask about this anyway – so may as well do it here. The point is not in whether data were cached or not, but really in a method to speed up the load process. I just confused caching and loading. I do not want to take away “best answer” from Spartacus, because he did answer what I asked ideally, thanks. But thank you very much Leonid for posting your answer too. +1 to both of you.

share|improve this question
    
It should be cached by default, I wonder why it doesn't happen on your system ... –  Szabolcs Feb 1 '12 at 18:39
    
@Szabolcs So if you restart M. now and type AstronomicalData[] you will not get brief "initializing indices" message? –  Vitaliy Kaurov Feb 1 '12 at 18:43
    
I get the initializing message, but it does not re-download. –  Szabolcs Feb 1 '12 at 18:49
2  
support.wolfram.com/kb/3310 –  Searke May 31 '12 at 14:07
add comment

2 Answers 2

up vote 13 down vote accepted

Initializing is not the same as downloading:

Mathematica graphics

I believe you are witnessing the data being unpacked for use.

share|improve this answer
    
I understand now, thanks. –  Vitaliy Kaurov Feb 1 '12 at 18:47
add comment

Please note that parts of the explanations and initialization code is shown together with the main function code, as a single large code block. I will appreciate any help on this matter - I am quite confused, perhaps overlooking something obvious here.


Preamble

While the question has been answered already, the delays with loading built-in data are a pretty serious problem, to my mind. This question prompted me to write a tiny framework, which will considerably speed it up, pretty much for arbitrary built-in data. The techniques involved will be a mix of memoization, some meta-programming, Block trick, and .mx files (DumpSave - Get).

The code

The first ingredient is the symbol-cloning functionality, which is described here (the function clone and related). I will reproduce this here to have a self-contained answer:

Clear[GlobalProperties];
GlobalProperties[] :=
  {OwnValues, DownValues, SubValues, UpValues, NValues, FormatValues, 
      Options, DefaultValues, Attributes};


Clear[unique];
unique[sym_] :=
 ToExpression[
    ToString[Unique[sym]] <> 
       StringReplace[StringJoin[ToString /@ Date[]], "." :> ""]];


Attributes[clone] = {HoldAll};
clone[s_Symbol, new_Symbol: Null] :=
  With[{clone = If[new === Null, unique[Unevaluated[s]], ClearAll[new]; new],
        sopts = Options[Unevaluated[s]]},
     With[{setProp = (#[clone] = (#[s] /. HoldPattern[s] :> clone)) &},
        Map[setProp, DeleteCases[GlobalProperties[], Options]];
        If[sopts =!= {}, Options[clone] = (sopts /. HoldPattern[s] :> clone)];
        HoldPattern[s] :> clone]]

Here are the functions of the "framework". The following is just a helper function to create a file name string:

ClearAll[makeFileName];
Options[makeFileName] = {
   TargetFileName :> Automatic,
   TargetDirectory :> $TemporaryDirectory
 };
makeFileName[dataFunction_Symbol, opts : OptionsPattern[]] :=
  With[{dir = OptionValue[TargetDirectory], fname = OptionValue[TargetFileName]},
      FileNameJoin[{
          dir, 
          If[fname =!= Automatic, 
             fname, 
             (* else *)
             "MemoizedData_" <> ToString[dataFunction] <> ".mx"
          ]}
      ]];

This is the main function. It will create dynamic environment (it returns a closure), inside which the values of a given data-holding function will be memoized. More details below.

ClearAll[generateMemoEnvironment];
SetAttributes[generateMemoEnvironment, HoldFirst];
Options[generateMemoEnvironment] = {
    StorageSymbol :> memoData,
    Sequence @@ Options[makeFileName]
 };
generateMemoEnvironment[env_Symbol, dataFunction_, opts : OptionsPattern[]] :=
   With[{memoSymbol = dataFunction /. clone[dataFunction],
      storageSymbol = OptionValue[StorageSymbol]
   },
   With[{fullname = makeFileName[ dataFunction, opts]},
       storageSymbol /: Save[storageSymbol[dataFunction]] :=
          DumpSave[fullname, {env, memoSymbol, storageSymbol}];
   ];
   env = 
     Function[
        code,
        Block[{dataFunction},
           dataFunction[args___] :=  storageSymbol[dataFunction][args];
           storageSymbol[dataFunction][args___] :=
               storageSymbol[dataFunction][args] = memoSymbol [args];
           code
        ],
        HoldAll]
];

How it works

What happens here is that, when we call generateMemoEnvironment, first the clone for a given symbol (e.g. for ChemicalData) is created (a clone is a symbol with identical global properties. Try f[x_]:=x;f[x_,y_]:=(x+y);clone[f,g] and look at definitions for g, to see what it does. The tricky point here is that the line dataFunction /. clone[dataFunction] calls dataFunction, which allows it to auto-load first. Otherwise, the clone would be empty. The main idea is that now, since we cloned the symbol, we can use Block to Block the main (original) symbol, and temporarily make it memoizing inside Block. Memoization is however done via an intermediate symbol memoSymbol. The main symbol which is kind of a "handle" for everything is given by the StorageSymbol option (I made it default to memoData) - it can be the same for all types of data.

Illustration and workflow

Let me now illustrate how to use this beast. Assume that you loaded the above code on a fresh kernel. Now, we create our dynamic environment:

generateMemoEnvironment[withMemoChemicalData, ChemicalData];

We can check that now the symbol withMemoChemicalData holds a pure function (closure):

 withMemoChemicalData

 (*
  ==>  Function[code$,Block[{ChemicalData},ChemicalData[args$___]:=
       memoData[ChemicalData][args$];memoData[ChemicalData][args$___]:=
      memoData[ChemicalData][args$]=ChemicalData$568201222222151718750[args$];
          code$],HoldAll]
 *)

Now, we execute some code within it (twice):

withMemoChemicalData[
   res1 = ChemicalData[#,"MolecularWeight"]&/@ChemicalData[]
];//Timing

(*
  ==> {20.375,Null}
*)

and again:

withMemoChemicalData[
    res2 = ChemicalData[#,"MolecularWeight"]&/@ChemicalData[]
];//Timing

(*
 ==> {0.125,Null}
*)

The timing difference reflects memoization at work. Note that, you can execute arbitrary code involving ChemicalData inside the environment, and memoization will work!

res1===res2

(*
 ==> True
*)

At the same time, because we used Block, the function ChemicalData did not receive any global definitions (which is easy to check) - which was one of the goals. This means, that our local modifications inside withMemoChemicalData present no danger whatsoever for the rest of the system, and / or other code which may be using ChemicalData from the outside of our environment.

To save the memoized values, you just call Save (I may get flamed for overloading it to work with a single argument, but that can be easily avoided if so desired):

Save[memoData[ChemicalData]];//Timing

(*
  ==> {0.454,Null}
*)

Now, here comes the main point: once you saved it once, you no longer need generateMemoEnvironment - you just need may be makeFileName, to construct the file name automatically. Let us now quit the kernel:

Quit

Now, we execute on a fresh kernel:

Get[makeFileName[ChemicalData]]

and we are ready to go:

withMemoChemicalData[
   res1 = ChemicalData[#,"MolecularWeight"]&/@ChemicalData[]
];//Timing


(*
 ==> {0.125,Null}
*)

Length[res1]

(*
 ==> 43987
*)

Moreover, you can now keep calling properties you did not call before, and those will be also memoized automatically - just wrap your code in withMemoChemicalData. All you have to do is to call Save[memoData[ChemicalData]] periodically, to update the file with newly memoized definitions.

Summary

I presented a tiny framework which may allow hundred-fold speed-ups when working with built-in data. The main ideas involved dynamic environments, metaprogramming, memoization, encapsulation, Block trick, and using .mx files to back up memoized values.

Comments and suggestions welcome!

share|improve this answer
    
Bug reported here: meta.stackexchange.com/questions/121004/… –  Szabolcs Feb 2 '12 at 0:31
    
@Szabolcs Thanks! It crossed my mind thatthe dollar sign might be a problem, but since there is no way for me to not use it, I did not test it. –  Leonid Shifrin Feb 2 '12 at 0:37
    
Perhaps it's best to leave the post in this broken state for a while so the SE folks can track the problem down and fix it. –  Szabolcs Feb 2 '12 at 0:39
    
@Szabolcs Yes, I also think so. This is actually the first time something like that happened in my practice. Did you encounter similar effects before? –  Leonid Shifrin Feb 2 '12 at 0:40
1  
@Leonid: impressive, as usual. I also think it could be really useful for practical use at least for rather 'static' data like ChemicalData. The main usage of FinancialData is to get data for time ranges and the caching of these might eat up a lot of memory and thus should probably be used with care. Might be true for other data functions like WeatherData as well. (I don't believe that FinancialData is locked because the developers did expect you to come along with this answer, though). It leaves a big question mark about why these built ins aren't more efficient in the first place... –  Albert Retey Jun 12 '12 at 22:16
show 11 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.