Repeated Calls to a Function that Imports Data the First Time

Question

I have several files with data in it that I need to use. I have written some code to import this data from one file, and then set up another function that depends on a list I read the data into to manipulate/recall the data.

I would like to generalize my importing function to be able to do the following:

1. Take as input a specific filename that I want imported
2. Perform the necessary importing (e.g. using the Import[] function) and manipulate the data to prepare it for use. For my case, this involves putting the imported data into a list.
3. Return a function that I can call which will be able to perform the necessary calculations to return the appropriate value from the list I imported.

The issue here is the variable I store the list in. As far as I can tell, either the list is a variable outside the scope of the function (a pain), or I need it to be a variable local to the function and I have to go through the process of reading the file every time I call the function (which is slow).

Is there a way to make a function basically "cache" some value, local to that function? I don't want to have to generate this list every time I call the function, nor do I want a bunch of extra variables lying around that I don't use.

Answer 1

Operator forms can be useful for cases such as this.

For purposes of illustration, we will start by defining a trivial import function. To keep this answer self-contained, our function will use ImportString instead of Import (but this entails no loss of generality):

import[dataString_] := ImportString[dataString, "CSV"] // First

import["10, 20, 30"]
(* {10, 20, 30} *)

Then, we will create an example calculation function:

calculate[data_, n_] := data + n

calculate[{10, 20, 30}, 2]
(* {12, 22, 32} *)

Finally, we will introduce an operator form of calculate, i.e. a function that produces another function that can be applied to the trailing argument n alone:

calculate[data_][n_] := calculate[data, n]

fn = calculate[{100, 200, 300}];
fn[3]
(* {103, 203, 303} *)
fn[4]
(* {104, 204, 304} *)

Armed with all this, the requirements from the question can be achieved:

fn = calculate[import["1000, 2000, 3000"]];
fn[5]
(* {1005, 2005, 3005} *)
fn[6]
(* {1006, 2006, 3006} *)

This strategy provides positive separation between the import and calculation concerns, and yet those separated concerns can easily be composed into a single function that imports, caches locally, and calculates.

The variable fn that is used in the examples above could be global or local. It is important to note that if fn is re-assigned, then the data that was previously cached will (eventually) be reclaimed. The cache does not leak out into any global symbols -- not even global symbols of the form x$123 which Mathematica uses to simulate local variables.

Answer 2

Create and export some random data,

Export["test.dat", {#, Sin[#]} & /@ Range[0, 6.1, .01]];

Then define your importing function, storing the data as a local variable inside an interpolating function, or just as a list whose elements we want access to

importFunction[fn_?FileExistsQ] := Module[{list},
   list = Import[fn, "Table"];
   Interpolation[list]
   ];

importFunction2[fn_?FileExistsQ] := Module[{list},
   list = Import[fn, "Table"];
   list[[#]] &
   ];

Now we can delete the file after we call the function just to show that we aren't repeatedly accessing the file,

testfun = importFunction["test.dat"];
testfun2 = importFunction2["test.dat"];

DeleteFile["test.dat"];
testfun[.17]
testfun2[18]
Sin[.17]
(* 0.169182 *)
(* {0.17, 0.169182} *)
(* 0.169182 *)

The list is stored as an internal variable, not really clogging up the namespace,

?testfun2

Global`testfun2

testfun2=list$5210[[#1]]&

You could also just use memoization to get the job done as well, then you won't need to define a new function for each file:

Export["test.dat", {#, Sin[#]} & /@ Range[0, 6.1, .01]];
importFunction3[fn_?FileExistsQ] := 
  importFunction3[fn] = Module[{list},
    list = Import[fn, "Table"];
    list[[#]] &
    ];
importFunction3["test.dat"][31]
DeleteFile["test.dat"];
importFunction3["test.dat"][23]
(* {0.3, 0.29552} *)
(* {0.22, 0.21823} *)

Again, I deleted the file after the first call just to show that it isn't accessed on subsequent calls