Resource management in Mathematica

Question

I am using a library that has functions like createSomeObject[] and deleteSomeObject[obj]. It is an interface to a C language library and it reflects how the C library is designed.

How can I make sure that when I have code similar to

Module[{obj},
  obj = createSomeObject[];
  (* some code here *)
  blackBoxFunction[];
  (* some code here *)
  deleteSomeObject[obj];
  (* ... *)
]

then the objects will always be safely cleaned up, even if the computation is aborted in the middle of the Module or blackBoxFunction[] does something unexpected (Throw[] or something else I didn't think of)?

In C++ we have RAII for this. What is the most usual / idiomatic way to do safe resource management this in Mathematica?

Note that even when only using built-in functions, this problem might come up with e.g. opening and closing file streams (it is very important to close streams). Even some packages shipping with Mathematica, such as TetGenLink, require explicit resource management.

---

Finally, is it possible to design such libraries in a way that avoids explicit deleteSomeObject[obj] calls? Is it possible to integrate the library with Mathematica's built-in reference counting and garbage collection? I found the "RemoveSymbol" handler (see Internal`Handlers[]), but it only seems to trigger on explicit Remove[], not when Temporary symbols (such as Module variables) are cleaned up.

---

Update: If you need this specifically for a LibraryLink library in Mathematica 10.0, look up Managed Library Expressions. This feature was not yet available when I wrote the question.

Answer 1

There is an undocumented function, CheckAll, that can be used for this purpose. It dates back to at least version 7. All the usual caveats about undocumented functions apply -- it might not be supported in future releases, there may be gaps in its functionality, etc. Buyer beware.

The usage information looks like this:

The usage text is slightly in error as the control arguments are wrapped in Hold rather than HoldComplete.

CheckAll can be used to detect all manner of non-local exits, such as:

CheckAll[Abort[], List]
(* {$Aborted, Hold[Abort[]]} *)

CheckAll[Throw[1], List]
(* {$Aborted, Hold[Throw[1]]} *)

CheckAll[Goto[a], List]
(* {$Aborted, Hold[Goto[a]]} *)

CheckAll[MemoryConstrained[Range@1000, 100], List]
(* {$Aborted, Hold[]} *)

CheckAll[TimeConstrained[Pause[1000], 1], List]
(* {$Aborted, Hold[]} *)

We can use this function to build unwindProtect, a control structure that assures that a clean-up expression is evaluated after any non-local exit of its body:

ClearAll @ unwindProtect
SetAttributes[unwindProtect, HoldAll]
unwindProtect[body_, cleanup_] :=
  CheckAll[body, HoldComplete] /.
    ( cleanup
    ; { _[_, _[r__]] :> r
      , _[r_, _[]] :> r
      }
    )

It starts by evaluating the body, guarded by CheckAll. Then it evaluates the clean-up expression. Finally, it returns either the pending non-local control actions or, if there are none, the return value of the body (which might be a held Sequence).

Here are some examples of its use:

hi[] := Print@"hi"
bye[] := Print@"bye"
fail[] := Print@"FAIL!"

unwindProtect[hi[]; Abort[], bye[]]
(* During evaluation of In[75]:= hi
   During evaluation of In[75]:= bye
   Out[75]= $Aborted *)

Catch @ unwindProtect[hi[]; Throw[1]; fail[], bye[]]
(* During evaluation of In[76]:= hi
   During evaluation of In[76]:= bye
   Out[76]= 1 *)

Module[{n = 0}
, Label[a]
; If[n < 2, unwindProtect[Print["hi ", ++n]; Goto[a], Print["bye ", n]]]
]
(* hi 1
   bye 1
   hi 2
   bye 2 *)

It even handles some tricky cases:

Catch@unwindProtect[hi[];Throw[Unevaluated[Abort[]]], bye[]]
(* During evaluation of In[82]:= hi
   During evaluation of In[82]:= bye
   Out[82]= $Aborted *)

Catch@unwindProtect[hi[];Throw[Unevaluated[Throw[$Failed]]], bye[]]
(* During evaluation of In[83]:= hi
   During evaluation of In[83]:= bye
   Out[83]= $Failed *)

unwindProtect can be used to build a still higher-level control structure (withSetup) that supports the declaration of Module-like variables, with sequential assignment and resource-cleanup expressions:

ClearAll[withSetup]
SetAttributes[withSetup, HoldAll]

withSetup[{}, body_] := body

withSetup[{var_ = val_; cleanup___, rest___}, body_] :=
  Module[{var = val}
  , unwindProtect[withSetup[{rest}, body], CompoundExpression[cleanup]]
  ]

withSetup[{var_ = val_, rest___}, body_] :=
  Module[{var = val}, withSetup[{rest}, body]]

w:withSetup[___] := (Message[withSetup::malformed, Short@HoldForm[w]]; Abort[])

withSetup::malformed = "``";

withSetup starts out resembling Module:

withSetup[{x = 1}, x + 1]
(* 2 *)

It differs from Module in that the variable assignments are performed sequentially:

withSetup[{x = 1, y = x + 1}, {x, y}]
(* {1, 2} *)

But the real value of withSetup is that it can be used to declare clean-up actions for each variable:

open[f_] := (Print["opened ", f]; file[f])
close[f_] := Print["closed ", f]

withSetup[{f = open["f1"]; close[f]}, f]
(* During evaluation of In[152]:= opened f1
   During evaluation of In[152]:= closed file[f1]
   Out[154]= file[f1] *)

... and those clean-up actions are performed even in the face of a non-local return:

Catch @ withSetup[{f = open["f1"]; close[f]}, Throw["early exit"]]
(* During evaluation of In[160]:= opened f1
   During evaluation of In[160]:= closed file[f1]
   Out[160]= early exit *)

Clean-up actions are performed in reverse order from their corresponding initializations:

withSetup[
  { file1 = open["f1"]; close[file1]
  , file2 = open["f2"]; close[file2]
  , files = {file1, file2}
  }
, doStuffWith[file1, file2, files]
]
(* During evaluation of In[155]:= opened f1
   During evaluation of In[155]:= opened f2
   During evaluation of In[155]:= closed file[f2]
   During evaluation of In[155]:= closed file[f1]
   doStuffWith[file[f1],file[f2],{file[f1],file[f2]}] *)

withSetup assumes that any variable initialization that is a compound expression is initialized using the first part of the expression, and is cleaned up using the remaining parts. One is free to write {var = (init1; init2); cleanup1; cleanup2} if desired.

Beware that any errors or other non-local exits from the clean-up actions can cause all kinds of strange behaviour. Clean-up actions should be foolproof, with no reasonable chance of failure. Wrap them in CheckAbort, AbortProtect or even unwindProtect if there is any doubt and circumstances demand it. unwindProtect does not do this automatically, although it could be changed to do so according to one's personal preference (I prefer to see the errors rather than muffle them).

Update

I subsequently discovered that CheckAll muffles all messages generated by the exit function. Furthermore, any messages from the main expression are also muffled in such circumstances. Here is a revised version of unwindProtect that performs some gymnastics to preserve the main messages and to inform the user when a clean-up expression fails:

ClearAll@unwindProtect
SetAttributes[unwindProtect, HoldAll]
unwindProtect[body_, cleanup_] :=
  CheckAll[body, HoldComplete] /.
    ( CheckAll[cleanup, HoldComplete] /. _[v_, _[c__]] :>
        Check[
          Message[unwindProtect::cleanupFailed
          , HoldForm @ cleanup
          , HoldForm @ {v}
          , HoldForm @ {c}
          ]
        , Null
        ]
    ; { _[_, _[r__]] :> r
      , _[r_, _[]] :> r
      }
    )

unwindProtect::cleanupFailed =
  "Cleanup expression failed: ``, results: ``, controls:  ``";

Answer 2

This question was indeed discussed on SO, here. I am usually using the version of CleanUp function by WReach, from that answer. It is however not fully bulletproof, as I noted in comments to that answer. Particular pieces of code which are problematic are nested exceptions or aborts like these:

Throw[Unevaluated[Abort[]]]

or

Throw[Unevaluated[Throw[$Failed]],

which are a problem because of the way untagged exceptions are caught (there is no way to intercept whatever is being thrown with them in unevaluated form). My personal opinion is that untagged exceptions are a language defect.

That said, CleanUp seems generally reliable, since the cases I described are rather unlikely, so I'd go ahead and use that (I do, actually). Generally, my feeling however is that there can be no totally proof reliable function to do that currently, and that this has to have a native support from the language (which is currently lacking). This view was expressed by WReach in his cited answer, and I share it.

There is also Internal`WithLocalSettings, however this one appears to not handle exceptions properly.

EDIT

Here is the setup I suggest. The code and an example follow.

Code

First, I reproduce here the CleanUp function from the cited answer by @WReach, to make this answer self-contained:

SetAttributes[CleanUp, HoldAll]
CleanUp[expr_, cleanup_] :=
  Module[{exprFn, result, abort = False, rethrow = True, seq}, 
    exprFn[] := expr;
    result = 
      CheckAbort[
         Catch[Catch[result = exprFn[]; rethrow = False; result], _, 
           seq[##] &], abort = True];
    cleanup;
    If[abort, Abort[]];
    If[rethrow, Throw[result /. seq -> Sequence]];
    result]

This is a custom Module, which would remove its variables explicitly. Note that it uses a custom remove function.

ClearAll[removingModule];
SetAttributes[removingModule, HoldAll];
removingModule[vars_, body_] :=
  Module[vars,
    CleanUp[
      body,
      ReleaseHold@Replace[
         HoldComplete[vars],
         {HoldPattern[a_Symbol = rhs_] :> remove[a], 
             HoldPattern[a_Symbol] :> remove[a]},
         {2}
      ]]];

Here is a custom remove function

ClearAll[remove, $removeHandler];
SetAttributes[remove, HoldFirst];
remove[s_Symbol, removeHandler_: $removeHandler ] :=
     CleanUp[$removeHandler[s], Remove[s]];
$removeHandler = Identity;

This will allow us to use familiar Module syntax in definitions (perhaps, incomplete, but illustrates the idea):

(* Note: there may be other rules added here *)
ClearAll[lex];
lex /: SetDelayed[lex@def_, rhs_] :=
   SetDelayed @@ 
     Prepend[Hold[rhs] /. Module -> removingModule, Unevaluated[def]];

This will generate custom dynamic environments, with a given handler which is invoked on remove:

ClearAll[withRemoveHandler];
withRemoveHandler[handler_] :=
  Function[code,
    Block[{$removeHandler  = handler}, code],
    HoldAll];

Example

We first define some function with our syntax:

ClearAll[myFunction];
lex@myFunction[x_] := Module[{y = 1, z = 2}, x + y + z];

Test:

?myFunction

Global`myFunction
myFunction[x_]:=removingModule[{y = 1, z = 2}, x+y+z]

Now, make a custom environment, where we will print the symbols before they get Remove-d:

env = withRemoveHandler[Function[s, Print[Unevaluated[s]], HoldAll]];

Now, execute code inside this environment:

In[98]:= env@myFunction[1]

During evaluation of In[98]:= y$705
During evaluation of In[98]:= z$705

Out[98]= 4

Summary of the approach

The above code may allow one to automate handling of resources in situations described in the original question (constructs like Module[{obj}, obj = createCustomObject[], ...]), while using the syntax maximally close to the standard one. You have to prefix your definitions with lex@, and execute code in a dynamic environment where you enable a handler that you want.

Answer 3

I would use the Execute Around Block pattern, which is superficially similar to RAII in c++. A simple example is what I use for ensuring that streams are closed after execution:

OpenAndRead[file_String, fcn_]:=
Module[{strm, res},
  strm = OpenRead[file];
  res = CheckAbort[ fcn[strm], $Aborted ];
  Close[strm];
  If[res === $Aborted, Abort[], res] (* Edited to allow Abort to propagate *)
]

Here, I use CheckAbort to prevent Abort from propagating out of my code until it I've closed the stream. It's use is simply,

fcn[ file_String, <otherparams> ] := OpenAndRead[file, fcn[#, <otherparams>]&]
fcn[ file_InputStream, <otherparams> ] := <fcn body>

But, that is inconvenient to use. In the previous answer, I suggested a facility for automatically generating that code. Since then, I've determined that is a poor strategy, and instead suggest the creation of a Block or Module type scoping construct:

SetAttributes[ReadBlock, HoldAll];
ReadBlock[filename_String, body_]:= ReadBlock[{filename, file}, body]

ReadBlock[{filename_String,file_Symbol}, body_] :=
 OpenAndRead[filename, Function[{file}, body]]

Then, the definition of fcn is

fcn[ file_String, <otherparams> ] := ReadBlock[{file, strm}, <body]

---

For your case, I'd set it up like this

SetAttributes[ProtectedBlock, HoldAll];
ProtectedBlock[{openfcn_, closefcn_},body_]:= 
    ProtectedBlock[{obj, openfcn, closefcn}, body]

ProtectedBlock[{obj_Symbol, openfcn_, closefcn_}, body_]:=
 Module[{res, obj},
    obj = openfcn;
    res = Check[ CheckAbort[ body, $Aborted ];closefcn , $Failed ];

    If[ 
      res === $Aborted, Abort[],
      res
    ]

 ]

Obviously, this needs a bit of work to properly rethrow the messages suppressed by Check, but it should be along the lines of what you are looking for.

---

Edit: Here's the final form of ReadBlock which I've renamed as BlockStream.

ClearAll[BlockStream];
SetAttributes[BlockStream, HoldAll];
SyntaxInformation[BlockStream] = {"ArgumentsPattern"->{_, _, _.}, 
          "LocalVariables" -> {"Solve", {1,1}}};
BlockStream[filename_, body_]:= 
    BlockStream[{file, filename}, OpenRead, body]
BlockStream[{file_Symbol, filename_}, body_]:= 
    BlockStream[{file, filename}, OpenRead, body]
BlockStream[filename_, op:(OpenRead | OpenWrite|OpenAppend), body_]:= 
    BlockStream[{file, filename}, op, body]
BlockStream[{file_Symbol, filename_}, op:(OpenRead | OpenWrite|OpenAppend), body_]:=
 Block[{file = op[filename], res, f, $TagLess},
  If[file ===$Failed, Return[$Failed]];
  (* 
	Catch added in case there is an uncaught Throw which would bypass 
    the Close[strm]. But, Catch[expr, _, f] would not catch tagless Throw, 
    so made Throw have dummy tag, $TagLess,
    which is stripped prior to rethrowing.
  *)
  res = Internal`InheritedBlock[{Throw},
         Unprotect[Throw];
         Throw[value_] := Throw[value, $TagLess];
		 Catch[CheckAbort[body, $Aborted], _, f]
      ];
  Close[file];

  Which[
   res === $Aborted, Abort[],
   Head@res === f, If[Last@res===$TagLess, res = f[First@res] ]; 
                   Block[{f = Throw}, res],
   True, res
  ]
 ]

This works by executing body within an environment where the Stream is guaranteed to be closed regardless of what the code in body does. There are two main ways to use it, with or without specifying a symbol to associate with the stream. If a symbol is not passed in, via the BlockStream[{file, filename}, ...] syntax, the symbol file is set to the stream and is exposed to the code within body. A simple usage example is

BlockStream[{strm, FileNameJoin[{$TemporaryDirectory, "writefile"}]},
 OpenWrite,
 Write[strm, a^2, 1 + b^2]
 ]

where as you can see the symbol strm is accesible within the body of the function.

Note this converges in many ways with Leonid's answer, but it uses only a single Catch. To deal with a tagless Throw, I modified throws behavior to add the tag $TagLess whenever a tag isn't specified, and I deal with that later. This seems like the best way to ensure that whether it is tagged and tagless, Throw is always caught.

Answer 4

According to the documentation, the Managed Library Expressions feature helps with this when using LibraryLink.

It makes it possible to create C-side data structures (using LibraryLink) which are automatically freed when the associated Mathematica expression is no longer referenced.

Answer 5

Version 12.2 introduced WithCleanup, an experimental but documented function to ensure safe resource disposal.

The example from the question would look like this when using WithCleanup:

Module[{obj},
  obj = createSomeObject[];
  WithCleanup[
    (* some code here *)
    blackBoxFunction[];
    (* some code here *)
    ,
    deleteSomeObject[obj];
  ]
  (* ... *)
]

deleteSomeObject is guaranteed to be called if control enters the WithCleanup expression whether or not the main expression terminates normally (barring catastrophic system failure such as a kernel crash).

There is also a three-argument form WithCleanup[init, body, cleanup] which allows initializations to be specified separately from the main body in which case those initializations will be protected from user aborts. But note that the clean-up expression is called even if the initialization form fails so if the clean-up expression depends upon the successful completion of the initialization forms then the latter should be placed outside of the WithCleanup form as shown in the example.