Resource management in Mathematica

Question

I am using a library that has functions like createSomeObject[] and deleteSomeObject[obj]. It is an interface to a C language library and it reflects how the C library is designed.

How can I make sure that when I have code similar to

Module[{obj},
  obj = createSomeObject[];
  (* some code here *)
  blackBoxFunction[];
  (* some code here *)
  deleteSomeObject[obj];
  (* ... *)
]

then the objects will always be safely cleaned up, even if the computation is aborted in the middle of the Module or blackBoxFunction[] does something unexpected (Throw[] or something else I didn't think of)?

In C++ we have RAII for this. What is the most usual / idiomatic way to do safe resource management this in Mathematica?

Note that even when only using built-in functions, this problem might come up with e.g. opening and closing file streams (it is very important to close streams). Even some packages shipping with Mathematica, such as TetGenLink, require explicit resource management.

---

Finally, is it possible to design such libraries in a way that avoids explicit deleteSomeObject[obj] calls? Is it possible to integrate the library with Mathematica's built-in reference counting and garbage collection? I found the "RemoveSymbol" handler (see Internal`Handlers[]), but it only seems to trigger on explicit Remove[], not when Temporary symbols (such as Module variables) are cleaned up.

Answer 1

This question was indeed discussed on SO, here. I am usually using the version of CleanUp function by WReach, from that answer. It is however not fully bulletproof, as I noted in comments to that answer. Particular pieces of code which are problematic are nested exceptions or aborts like these:

Throw[Unevaluated[Abort[]]]

or

Throw[Unevaluated[Throw[$Failed]],

which are a problem because of the way untagged exceptions are caught (there is no way to intercept whatever is being thrown with them in unevaluated form). My personal opinion is that untagged exceptions are a language defect.

That said, CleanUp seems generally reliable, since the cases I described are rather unlikely, so I'd go ahead and use that (I do, actually). Generally, my feeling however is that there can be no totally proof reliable function to do that currently, and that this has to have a native support from the language (which is currently lacking). This view was expressed by WReach in his cited answer, and I share it.

There is also Internal`WithLocalSettings`, however this one appears to not handle exceptions properly.

EDIT

Here is the setup I suggest. The code and an example follow.

Code

First, I reproduce here the CleanUp function from the cited answer by @WReach, to make this answer self-contained:

SetAttributes[CleanUp, HoldAll]
CleanUp[expr_, cleanup_] :=
  Module[{exprFn, result, abort = False, rethrow = True, seq}, 
    exprFn[] := expr;
    result = 
      CheckAbort[
         Catch[Catch[result = exprFn[]; rethrow = False; result], _, 
           seq[##] &], abort = True];
    cleanup;
    If[abort, Abort[]];
    If[rethrow, Throw[result /. seq -> Sequence]];
    result]

This is a custom Module, which would remove its variables explicitly. Note that it uses a custom remove function.

ClearAll[removingModule];
SetAttributes[removingModule, HoldAll];
removingModule[vars_, body_] :=
  Module[vars,
    CleanUp[
      body,
      ReleaseHold@Replace[
         HoldComplete[vars],
         {HoldPattern[a_Symbol = rhs_] :> remove[a], 
             HoldPattern[a_Symbol] :> remove[a]},
         {2}
      ]]];

Here is a custom remove function

ClearAll[remove, $removeHandler];
SetAttributes[remove, HoldFirst];
remove[s_Symbol, removeHandler_: $removeHandler ] :=
     CleanUp[$removeHandler[s], Remove[s]];
$removeHandler = Identity;

This will allow us to use familiar Module syntax in definitions (perhaps, incomplete, but illustrates the idea):

(* Note: there may be other rules added here *)
ClearAll[lex];
lex /: SetDelayed[lex@def_, rhs_] :=
   SetDelayed @@ 
     Prepend[Hold[rhs] /. Module -> removingModule, Unevaluated[def]];

This will generate custom dynamic environments, with a given handler which is invoked on remove:

ClearAll[withRemoveHandler];
withRemoveHandler[handler_] :=
  Function[code,
    Block[{$removeHandler  = handler}, code],
    HoldAll];

Example

We first define some function with our syntax:

ClearAll[myFunction];
lex@myFunction[x_] := Module[{y = 1, z = 2}, x + y + z];

Test:

?myFunction

Global`myFunction
myFunction[x_]:=removingModule[{y = 1, z = 2}, x+y+z]

Now, make a custom environment, where we will print the symbols before they get Remove-d:

env = withRemoveHandler[Function[s, Print[Unevaluated[s]], HoldAll]];

Now, execute code inside this environment:

In[98]:= env@myFunction[1]

During evaluation of In[98]:= y$705
During evaluation of In[98]:= z$705

Out[98]= 4

Summary of the approach

The above code may allow one to automate handling of resources in situations described in the original question (constructs like Module[{obj}, obj = createCustomObject[], ...]), while using the syntax maximally close to the standard one. You have to prefix your definitions with lex@, and execute code in a dynamic environment where you enable a handler that you want.

Answer 2

I would use the Execute Around Block pattern, which is superficially similar to RAII in c++. A simple example is what I use for ensuring that streams are closed after execution:

OpenAndRead[file_String, fcn_]:=
Module[{strm, res},
  strm = OpenRead[file];
  res = CheckAbort[ fcn[strm], $Aborted ];
      Close[strm];
      If[res === $Aborted, Abort[], res] (* Edited to allow Abort to propagate *)
]

Here, I use CheckAbort to prevent Abort from propagating out of my code until it I've closed the stream. It's use is simply,

fcn[ file_String, <otherparams> ] := OpenAndRead[file, fcn[#, <otherparams>]&]
fcn[ file_InputStream, <otherparams> ] := <fcn body>

But, that is inconvenient to use. In the previous answer, I suggested a facility for automatically generating that code. Since then, I've determined that is a poor strategy, and instead suggest the creation of a Block or Module type scoping construct:

SetAttributes[ReadBlock, HoldAll];
ReadBlock[filename_String, body_]:= ReadBlock[{filename, file}, body]

ReadBlock[{filename_String,file_Symbol}, body_] :=
 OpenAndRead[filename, Function[{file}, body]]

Then, the definition of fcn is

fcn[ file_String, <otherparams> ] := ReadBlock[{file, strm}, <body]

---

For your case, I'd set it up like this

SetAttributes[ProtectedBlock, HoldAll];
ProtectedBlock[{openfcn_, closefcn_},body_]:= 
    ProtectedBlock[{obj, openfcn, closefcn}, body]

ProtectedBlock[{obj_Symbol, openfcn_, closefcn_}, body_]:=
 Module[{res, obj},
    obj = openfcn;
    res = Check[ CheckAbort[ body, $Aborted ];closefcn , $Failed ];

    If[ 
      res === $Aborted, Abort[],
      res
    ]

 ]

Obviously, this needs a bit of work to properly rethrow the messages suppressed by Check, but it should be along the lines of what you are looking for.

---

Edit: Here's the final form of ReadBlock which I've renamed as BlockStream.

ClearAll[BlockStream];
SetAttributes[BlockStream, HoldAll];
SyntaxInformation[BlockStream] = {"ArgumentsPattern"->{_, _, _.}, 
          "LocalVariables" -> {"Solve", {1,1}}};
BlockStream[filename_, body_]:= 
    BlockStream[{file, filename}, OpenRead, body]
BlockStream[{file_Symbol, filename_}, body_]:= 
    BlockStream[{file, filename}, OpenRead, body]
BlockStream[filename_, op:(OpenRead | OpenWrite|OpenAppend), body_]:= 
    BlockStream[{file, filename}, op, body]
BlockStream[{file_Symbol, filename_}, op:(OpenRead | OpenWrite|OpenAppend), body_]:=
 Block[{file = op[filename], res, f, $TagLess},
      If[file ===$Failed, Return[$Failed]];
      (* 
    	Catch added in case there is an uncaught Throw which would bypass 
        the Close[strm]. But, Catch[expr, _, f] would not catch tagless Throw, 
        so made Throw have dummy tag, $TagLess,
    which is stripped prior to rethrowing.
  *)
  res = Internal`InheritedBlock[{Throw},
         Unprotect[Throw];
         Throw[value_] := Throw[value, $TagLess];
    		 Catch[CheckAbort[body, $Aborted], _, f]
      ];
  Close[file];

  Which[
   res === $Aborted, Abort[],
       Head@res === f, If[Last@res===$TagLess, res = f[First@res] ]; 
                   Block[{f = Throw}, res],
   True, res
  ]
 ]

This works by executing body within an environment where the Stream is guaranteed to be closed regardless of what the code in body does. There are two main ways to use it, with or without specifying a symbol to associate with the stream. If a symbol is not passed in, via the BlockStream[{file, filename}, ...] syntax, the symbol file is set to the stream and is exposed to the code within body. A simple usage example is

BlockStream[{strm, FileNameJoin[{$TemporaryDirectory, "writefile"}]},
 OpenWrite,
 Write[strm, a^2, 1 + b^2]
 ]

where as you can see the symbol strm is accesible within the body of the function.

Note this converges in many ways with Leonid's answer, but it uses only a single Catch. To deal with a tagless Throw, I modified throws behavior to add the tag $TagLess whenever a tag isn't specified, and I deal with that later. This seems like the best way to ensure that whether it is tagged and tagless, Throw is always caught.