22
$\begingroup$

There are Mathematica packages that must create temporary files to function. If we are implementing such a package ourselves, how can we ensure that the temporary files will get cleaned up when the kernel exits?


Is this really a practical problem? Yes, both MATLink and MaTeX need to do it. Neither are able to do a full cleanup at the moment. Standard packages do it too, e.g. CCompilerDriver`.

Is it really possible to do it? Yes. Compile creates shared libraries when compiling to C code. These do get cleaned up on exit. How is this implemented?

In[1]:= cf = Compile[{{x}}, 2 x, CompilationTarget -> "C"];

In[2]:= FileNames[
 FileNameJoin[{$UserBaseDirectory, "ApplicationData", 
   "CCompilerDriver", "BuildFolder", "*", "*"}]]

Out[2]= {"/Users/szhorvat/Library/Mathematica/ApplicationData/\
CCompilerDriver/BuildFolder/hawkeye-8727/compiledFunction0.dylib"}

In[3]:= Quit

In[1]:= FileNames[
 FileNameJoin[{$UserBaseDirectory, "ApplicationData", 
   "CCompilerDriver", "BuildFolder", "*", "*"}]]

Out[1]= {}

We can use $Epilog for this! $Epilog is by default defined to evaluate << end`. This file does not exist by default (it is not found by FindFile). One possible solution is to create this file and add the cleanup code to it at the time when the package loads. But it is not clear how to this in a robust way, without conflicting with other packages that might also modify the same file.

The cleanup mechanism for shared libraries created by Compile does not seem to use this. Thus I am still hoping to find a better and more robust solution. What does Compile use?


One way to try to find out what happens when existing the kernel is to evaluate On[], followed by Quit[]. This is best done in a terminal session, which does not need to deal with front-end interaction. This trick does not reveal any other evaluations that the one triggered by $Epilog. The default value of $Epilog in Mathematica 10 is If[FindFile["end`"] =!= $Failed, Get["end`"]].

$\endgroup$
  • 3
    $\begingroup$ Silly question: what about cleaning at start? Less silly question: isn't there any way to deal with this on OS level, like attaching a 'listener' to the process that will do this when the process is terminated? $\endgroup$ – Kuba Jan 19 '16 at 10:46
  • $\begingroup$ @Kuba To do it at the OS level is a good idea! It can be a separate process communicating through MathLink, or even simpler: a LibraryLink library which does the cleanup on "uninitialization". I would still like to know how the Compile stuff achieves it though. $\endgroup$ – Szabolcs Jan 19 '16 at 10:55
  • $\begingroup$ @Kuba I realize I checked the value of $Epilog incorrectly. It does have a value, it will do <<end` if such a file exists. It is not what cleans it up though. What I tried now was to evaluate On[] before exit (in a terminal, no front end!) to see what gets evaluated. I'm trying to sort through that ... $\endgroup$ – Szabolcs Jan 19 '16 at 11:12
  • $\begingroup$ I am interested in an answer to this, and in particular your solution by including cleanup code in WolframLibrary_uninitialize. Could you post a sample code that cleans up a few known files? $\endgroup$ – QuantumDot Jul 8 '17 at 5:09
  • $\begingroup$ @QuantumDot I'm still travelling for a week, can you remind me after that please? $\endgroup$ – Szabolcs Jul 12 '17 at 5:20
7
$\begingroup$

I have found $Epilog utterly unreliable for this. Not all ways of quitting Mathematica evaluates $Epilog. And of course, there is the problem that a different package may (re)set it.

I have no idea how Compile implements this, but I have two suggestions:

  1. Place all your temporary files in a special folder that you empty on load. Be careful how you handle parallel kernels that may load multiple versions of your package at the same or interleaving times.

  2. Write a small MathLink program that you Install on startup, has a function AddFileForDeletion that adds the path for a file to an internal list (in the C/C++ world), which it then iterates over and deletes after losing the MathLink connection.

Suggestion 2 is the most reliable way I've found to do temporary file cleanup.

$\endgroup$
  • $\begingroup$ Thanks! I was also thinking of doing something like 2., but instead of a MathLink program it is also possible to use a LibraryLink library (and put the cleanup code in WolframLibrary_uninitialize. Would it at all help if I sent an email to support and suggested to add a reliable builtin cleanup mechanism? $\endgroup$ – Szabolcs Apr 21 '16 at 7:40
  • 1
    $\begingroup$ I certainly agree, it's ridiculous to have to go to such lengths for something as simple as this. $\endgroup$ – Malte Lenz Apr 21 '16 at 8:45
  • 5
    $\begingroup$ I think it's unlikely the LibraryLink proposal would work. The FE kills the kernel by sending MLTerminateMessage, which triggers a signal handler, which promptly exits. The advantage of this is that it always works promptly, no matter how hard the kernel might be hung. The disadvantage is that signal handlers are very limiting. You can't even call malloc or free in one, and definitely file system calls are right out. I would be surprised if the signal handler calls LibraryLink uninitialize functions. That having been said, I'm not an expert in the kernel functionality here. $\endgroup$ – John Fultz Jul 9 '17 at 5:28
  • 5
    $\begingroup$ That having been said, it's not a ridiculous suggestion that the FE ought to try to get the kernel to quit on its own, and only terminate it if it fails to do so after a brief timeout. That would enable end.m, WolframLibrary_uninitialize, etc., to run much more frequently...but there would still be no guarantees. $\endgroup$ – John Fultz Jul 9 '17 at 5:39
  • 1
    $\begingroup$ @Szabolcs I've not looked at the implementation for LibraryLink on this. I'm surprised by your results (but not surprised that you're running into limitations), though, and I'm very confident about my statements regarding signal handling and MLTerminateMessage. In fact, I recently fixed a bug in WSTP/MathLink for 11.2 which could cause a zombie kernel because MathLink was...ahem...calling malloc inside of a signal handler. Under Linux, the MathLink signal handler was getting invoked sometimes inside of the kernel calling malloc, and it hung hard. $\endgroup$ – John Fultz Jul 31 '17 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.