Resource management in Mathematica

I am using a library that has functions like createSomeObject[] and deleteSomeObject[obj]. It is an interface to a C language library and it reflects how the C library is designed.

How can I make sure that when I have code similar to

Module[{obj},
obj = createSomeObject[];
(* some code here *)
blackBoxFunction[];
(* some code here *)
deleteSomeObject[obj];
(* ... *)
]


then the objects will always be safely cleaned up, even if the computation is aborted in the middle of the Module or blackBoxFunction[] does something unexpected (Throw[] or something else I didn't think of)?

In C++ we have RAII for this. What is the most usual / idiomatic way to do safe resource management this in Mathematica?

Note that even when only using built-in functions, this problem might come up with e.g. opening and closing file streams (it is very important to close streams). Even some packages shipping with Mathematica, such as TetGenLink, require explicit resource management.

Finally, is it possible to design such libraries in a way that avoids explicit deleteSomeObject[obj] calls? Is it possible to integrate the library with Mathematica's built-in reference counting and garbage collection? I found the "RemoveSymbol" handler (see InternalHandlers[]), but it only seems to trigger on explicit Remove[], not when Temporary symbols (such as Module variables) are cleaned up.

-
I am sure I read a similar question on StackOverflow, but I can't find it now ... –  Szabolcs Feb 10 '12 at 17:08

This question was indeed discussed on SO, here. I am usually using the version of CleanUp function by WReach, from that answer. It is however not fully bulletproof, as I noted in comments to that answer. Particular pieces of code which are problematic are nested exceptions or aborts like these:

Throw[Unevaluated[Abort[]]]


or

Throw[Unevaluated[Throw[$Failed]],  which are a problem because of the way untagged exceptions are caught (there is no way to intercept whatever is being thrown with them in unevaluated form). My personal opinion is that untagged exceptions are a language defect. That said, CleanUp seems generally reliable, since the cases I described are rather unlikely, so I'd go ahead and use that (I do, actually). Generally, my feeling however is that there can be no totally proof reliable function to do that currently, and that this has to have a native support from the language (which is currently lacking). This view was expressed by WReach in his cited answer, and I share it. There is also InternalWithLocalSettings, however this one appears to not handle exceptions properly. EDIT Here is the setup I suggest. The code and an example follow. Code First, I reproduce here the CleanUp function from the cited answer by @WReach, to make this answer self-contained: SetAttributes[CleanUp, HoldAll] CleanUp[expr_, cleanup_] := Module[{exprFn, result, abort = False, rethrow = True, seq}, exprFn[] := expr; result = CheckAbort[ Catch[Catch[result = exprFn[]; rethrow = False; result], _, seq[##] &], abort = True]; cleanup; If[abort, Abort[]]; If[rethrow, Throw[result /. seq -> Sequence]]; result]  This is a custom Module, which would remove its variables explicitly. Note that it uses a custom remove function. ClearAll[removingModule]; SetAttributes[removingModule, HoldAll]; removingModule[vars_, body_] := Module[vars, CleanUp[ body, ReleaseHold@Replace[ HoldComplete[vars], {HoldPattern[a_Symbol = rhs_] :> remove[a], HoldPattern[a_Symbol] :> remove[a]}, {2} ]]];  Here is a custom remove function ClearAll[remove,$removeHandler];
SetAttributes[remove, HoldFirst];
remove[s_Symbol, removeHandler_: $removeHandler ] := CleanUp[$removeHandler[s], Remove[s]];
$removeHandler = Identity;  This will allow us to use familiar Module syntax in definitions (perhaps, incomplete, but illustrates the idea): (* Note: there may be other rules added here *) ClearAll[lex]; lex /: SetDelayed[lex@def_, rhs_] := SetDelayed @@ Prepend[Hold[rhs] /. Module -> removingModule, Unevaluated[def]];  This will generate custom dynamic environments, with a given handler which is invoked on remove: ClearAll[withRemoveHandler]; withRemoveHandler[handler_] := Function[code, Block[{$removeHandler  = handler}, code],
HoldAll];


Example

We first define some function with our syntax:

ClearAll[myFunction];
lex@myFunction[x_] := Module[{y = 1, z = 2}, x + y + z];


Test:

?myFunction

GlobalmyFunction
myFunction[x_]:=removingModule[{y = 1, z = 2}, x+y+z]


Now, make a custom environment, where we will print the symbols before they get Remove-d:

env = withRemoveHandler[Function[s, Print[Unevaluated[s]], HoldAll]];


Now, execute code inside this environment:

In[98]:= env@myFunction[1]

During evaluation of In[98]:= y$705 During evaluation of In[98]:= z$705

Out[98]= 4


Summary of the approach

The above code may allow one to automate handling of resources in situations described in the original question (constructs like Module[{obj}, obj = createCustomObject[], ...]), while using the syntax maximally close to the standard one. You have to prefix your definitions with lex@, and execute code in a dynamic environment where you enable a handler that you want.

-
Do you have some more information on InternalWithLocalSettings ? It seems to be just what I asked for. How does it fail? In a quick test I did notice that it sometimes, but not always prevents exceptions from being caught, which is very weird behaviour: Catch[ InternalWithLocalSettings[Print["begin"], Throw[1]; Print["not printed"], Print["end"]] ] doesn't work correctly, but Catch[ InternalWithLocalSettings[Print["begin"], Throw[1]; Print["not printed"], nothing] ] does. –  Szabolcs Feb 10 '12 at 17:32
@Szabolcs Unfortunately I don't. It looks like InternalWithLocalSettings has its own way to unwind the stack, which conflicts with Catch. I don't really know how it works. One thing which is strange is that in the first case (when it leaks an exception), it returns $Aborted[], meaning that some Abort[] is happening somewhere. It could be that it sends 3 different evaluations to the kernel via MathLink, and therefore, even when an exception is happening, we have no way to catch it. Why and what gets Aborted I have no idea. – Leonid Shifrin Feb 10 '12 at 17:55 I am still wishing for some kind of integration with the garbage collector ... then a plain Module would do – Szabolcs Feb 10 '12 at 18:00 @Szabolcs Yes, I agree. I will actually delete the above comment, did not put much thinking in it (seems to be my problem for the past few days). You could use your own scoping construct, call it myModule, and get some similar structure there - this should be safe. – Leonid Shifrin Feb 10 '12 at 18:37 @Szabolcs I added some code which may address some of your concerns. – Leonid Shifrin Feb 10 '12 at 20:03 I would use the Execute Around Block pattern, which is superficially similar to RAII in c++. A simple example is what I use for ensuring that streams are closed after execution: OpenAndRead[file_String, fcn_]:= Module[{strm, res}, strm = OpenRead[file]; res = CheckAbort[ fcn[strm],$Aborted ];
Close[strm];
If[res === $Aborted, Abort[], res] (* Edited to allow Abort to propagate *) ]  Here, I use CheckAbort to prevent Abort from propagating out of my code until it I've closed the stream. It's use is simply, fcn[ file_String, <otherparams> ] := OpenAndRead[file, fcn[#, <otherparams>]&] fcn[ file_InputStream, <otherparams> ] := <fcn body>  But, that is inconvenient to use. In the previous answer, I suggested a facility for automatically generating that code. Since then, I've determined that is a poor strategy, and instead suggest the creation of a Block or Module type scoping construct: SetAttributes[ReadBlock, HoldAll]; ReadBlock[filename_String, body_]:= ReadBlock[{filename, file}, body] ReadBlock[{filename_String,file_Symbol}, body_] := OpenAndRead[filename, Function[{file}, body]]  Then, the definition of fcn is fcn[ file_String, <otherparams> ] := ReadBlock[{file, strm}, <body]  For your case, I'd set it up like this SetAttributes[ProtectedBlock, HoldAll]; ProtectedBlock[{openfcn_, closefcn_},body_]:= ProtectedBlock[{obj, openfcn, closefcn}, body] ProtectedBlock[{obj_Symbol, openfcn_, closefcn_}, body_]:= Module[{res, obj}, obj = openfcn; res = Check[ CheckAbort[ body,$Aborted ];closefcn , $Failed ]; If[ res ===$Aborted, Abort[],
res
]

]


Obviously, this needs a bit of work to properly rethrow the messages suppressed by Check, but it should be along the lines of what you are looking for.

Edit: Here's the final form of ReadBlock which I've renamed as BlockStream.

ClearAll[BlockStream];
SetAttributes[BlockStream, HoldAll];
SyntaxInformation[BlockStream] = {"ArgumentsPattern"->{_, _, _.},
"LocalVariables" -> {"Solve", {1,1}}};
BlockStream[filename_, body_]:=
BlockStream[{file_Symbol, filename_}, body_]:=
BlockStream[{file, filename}, op, body]
BlockStream[{file_Symbol, filename_}, op:(OpenRead | OpenWrite|OpenAppend), body_]:=
Block[{file = op[filename], res, f, $TagLess}, If[file ===$Failed, Return[$Failed]]; (* Catch added in case there is an uncaught Throw which would bypass the Close[strm]. But, Catch[expr, _, f] would not catch tagless Throw, so made Throw have dummy tag,$TagLess,
which is stripped prior to rethrowing.
*)
res = InternalInheritedBlock[{Throw},
Unprotect[Throw];
Throw[value_] := Throw[value, $TagLess]; Catch[CheckAbort[body,$Aborted], _, f]
];
Close[file];

Which[
res === $Aborted, Abort[], Head@res === f, If[Last@res===$TagLess, res = f[First@res] ];
Block[{f = Throw}, res],
True, res
]
]


This works by executing body within an environment where the Stream is guaranteed to be closed regardless of what the code in body does. There are two main ways to use it, with or without specifying a symbol to associate with the stream. If a symbol is not passed in, via the BlockStream[{file, filename}, ...] syntax, the symbol file is set to the stream and is exposed to the code within body. A simple usage example is

BlockStream[{strm, FileNameJoin[{$TemporaryDirectory, "writefile"}]}, OpenWrite, Write[strm, a^2, 1 + b^2] ]  where as you can see the symbol strm is accesible within the body of the function. Note this converges in many ways with Leonid's answer, but it uses only a single Catch. To deal with a tagless Throw, I modified throws behavior to add the tag $TagLess whenever a tag isn't specified, and I deal with that later. This seems like the best way to ensure that whether it is tagged and tagless, Throw is always caught.

-
One problem I find with this is that this does not handle exceptions. And when you include exceptions, I think you are likely to end up with someting like CleanUp, for which I can not claim the authorship alas :) –  Leonid Shifrin Feb 10 '12 at 18:44
@LeonidShifrin you're right it doesn't handle exceptions. But, I find that if an exception escapes into my code from a library, then the library isn't well written. But, it could be adapted to do that. And, I had not actually read the code for CleanUp before, and it is very similar. –  rcollyer Feb 10 '12 at 18:50
I am not sure regarding exceptions. Java has exceptions and most libraries throw some to the user in some cases, including run-time exceptions which are not checked by the compiler. Exceptions are the signal that something should go through the execution stack until it meets the proper handler. It often happens that your library is not the end of the stack, so you should simply pass it along, to let the user of your library handle that. Unless your code must handle them, you should allow them to pass along, performing proper actions, rather than have your code broken because of them ... –  Leonid Shifrin Feb 10 '12 at 19:01
... What I described is a common wisdom for large-scale software engineering say in Java. OTOH, in Mathematica exceptions are not widely used, and returning $Failed is preferred. I attribute that to the fact that for Mathematica applications (packages) large technology stacks are not typical, because large third-party Mathematica applications as components are not typical. As soon as larger body of Mathematica code available as components / frameworks accumulates, this may change. Besides, if you write a large app, you may wish some of your modules to throw exceptions. – Leonid Shifrin Feb 10 '12 at 19:06 Yes, sure, but you (as the one who supplies a general function such as ProtectedBlock) should aim at module developers more than users, IMO. Because anyone who needs to use something like CleanUp or ProtectedBlock, better be considered a developer. One should make tools that one would like to use him/herself, not dumbed-down tools for hypothetical end users who must be protected from complexity. Such users get UI-s and final simple interfaces, behind which everything is hidden, and will never need CleanUp. This is not my thought (alas), but I fully agree with it. – Leonid Shifrin Feb 10 '12 at 20:26 There is an undocumented function, CheckAll, that can be used for this purpose. It dates back to at least version 7. All the usual caveats about undocumented functions apply -- it might not be supported in future releases, there may be gaps in its functionality, etc. Buyer beware. The usage information looks like this: The usage text is slightly in error as the control arguments are wrapped in Hold rather than HoldComplete. CheckAll can be used to detect all manner of non-local exits, such as: CheckAll[Abort[], List] (* {$Aborted, Hold[Abort[]]} *)

CheckAll[Throw[1], List]
(* {$Aborted, Hold[Throw[1]]} *) CheckAll[Goto[a], List] (* {$Aborted, Hold[Goto[a]]} *)

CheckAll[MemoryConstrained[Range@1000, 100], List]
(* {$Aborted, Hold[]} *) CheckAll[TimeConstrained[Pause[1000], 1], List] (* {$Aborted, Hold[]} *)


We can use this function to build unwindProtect, a control structure that assures that a clean-up expression is evaluated after any non-local exit of its body:

ClearAll @ unwindProtect
SetAttributes[unwindProtect, HoldAll]
unwindProtect[body_, cleanup_] :=
CheckAll[body, HoldComplete] /.
( cleanup
; { _[_, _[r__]] :> r
, _[r_, _[]] :> r
}
)


It starts by evaluating the body, guarded by CheckAll. Then it evaluates the clean-up expression. Finally, it returns either the pending non-local control actions or, if there are none, the return value of the body (which might be a held Sequence).

Here are some examples of its use:

hi[] := Print@"hi"
bye[] := Print@"bye"
fail[] := Print@"FAIL!"

unwindProtect[hi[]; Abort[], bye[]]
(* During evaluation of In[75]:= hi
During evaluation of In[75]:= bye
Out[75]= $Aborted *) Catch @ unwindProtect[hi[]; Throw[1]; fail[], bye[]] (* During evaluation of In[76]:= hi During evaluation of In[76]:= bye Out[76]= 1 *) Module[{n = 0} , Label[a] ; If[n < 2, unwindProtect[Print["hi ", ++n]; Goto[a], Print["bye ", n]]] ] (* hi 1 bye 1 hi 2 bye 2 *)  It even handles some tricky cases: Catch@unwindProtect[hi[];Throw[Unevaluated[Abort[]]], bye[]] (* During evaluation of In[82]:= hi During evaluation of In[82]:= bye Out[82]=$Aborted *)

Catch@unwindProtect[hi[];Throw[Unevaluated[Throw[$Failed]]], bye[]] (* During evaluation of In[83]:= hi During evaluation of In[83]:= bye Out[83]=$Failed *)


unwindProtect can be used to build a still higher-level control structure (withSetup) that supports the declaration of Module-like variables, with sequential assignment and resource-cleanup expressions:

ClearAll[withSetup]
SetAttributes[withSetup, HoldAll]

withSetup[{}, body_] := body

withSetup[{var_ = val_; cleanup___, rest___}, body_] :=
Module[{var = val}
, unwindProtect[withSetup[{rest}, body], CompoundExpression[cleanup]]
]

withSetup[{var_ = val_, rest___}, body_] :=
Module[{var = val}, withSetup[{rest}, body]]

w:withSetup[___] := (Message[withSetup::malformed, Short@HoldForm[w]]; Abort[])

withSetup::malformed = "";


withSetup starts out resembling Module:

withSetup[{x = 1}, x + 1]
(* 2 *)


It differs from Module in that the variable assignments are performed sequentially:

withSetup[{x = 1, y = x + 1}, {x, y}]
(* {1, 2} *)


But the real value of withSetup is that it can be used to declare clean-up actions for each variable:

open[f_] := (Print["opened ", f]; file[f])
close[f_] := Print["closed ", f]

withSetup[{f = open["f1"]; close[f]}, f]
(* During evaluation of In[152]:= opened f1
During evaluation of In[152]:= closed file[f1]
Out[154]= file[f1] *)


... and those clean-up actions are performed even in the face of a non-local return:

Catch @ withSetup[{f = open["f1"]; close[f]}, Throw["early exit"]]
(* During evaluation of In[160]:= opened f1
During evaluation of In[160]:= closed file[f1]
Out[160]= early exit *)


Clean-up actions are performed in reverse order from their corresponding initializations:

withSetup[
{ file1 = open["f1"]; close[file1]
, file2 = open["f2"]; close[file2]
, files = {file1, file2}
}
, doStuffWith[file1, file2, files]
]
(* During evaluation of In[155]:= opened f1
During evaluation of In[155]:= opened f2
During evaluation of In[155]:= closed file[f2]
During evaluation of In[155]:= closed file[f1]
doStuffWith[file[f1],file[f2],{file[f1],file[f2]}] *)


withSetup assumes that any variable initialization that is a compound expression expression is initialized using the first part of the expression, and is cleaned up using the remaining parts. One is free to write {var = (init1; init2); cleanup1; cleanup2} if desired.

Beware that any errors or other non-local exits from the clean-up actions can cause all kinds of strange behaviour. Clean-up actions should be foolproof, with no reasonable chance of failure. Wrap them in CheckAbort, AbortProtect or even unwindProtect if there is any doubt and circumstances demand it. unwindProtect does not do this automatically, although it could be changed to do so according to one's personal preference (I prefer to see the errors rather than muffle them).

Update

I subsequently discovered that CheckAll muffles all messages generated by the exit function. Furthermore, any messages from the main expression are also muffled in such circumstances. Here is a revised version of unwindProtect that performs some gymnastics to preserve the main messages and to inform the user when a clean-up expression fails:

ClearAll@unwindProtect
SetAttributes[unwindProtect, HoldAll]
unwindProtect[body_, cleanup_] :=
CheckAll[body, HoldComplete] /.
( CheckAll[cleanup, HoldComplete] /. _[v_, _[c__]] :>
Check[
Message[unwindProtect::cleanupFailed
, HoldForm @ cleanup
, HoldForm @ {v}
, HoldForm @ {c}
]
, Null
]
; { _[_, _[r__]] :> r
, _[r_, _[]] :> r
}
)

unwindProtect::cleanupFailed =
"Cleanup expression failed: , results: , controls:  ";
`
-
Wow, great find! –  Szabolcs May 25 at 0:42

According to the documentation, the Managed Library Expressions feature helps with this when using LibraryLink.

-