22
$\begingroup$

There are Mathematica packages that must create temporary files to function. If we are implementing such a package ourselves, how can we ensure that the temporary files will get cleaned up when the kernel exits?

Is this really a practical problem? Yes, both MATLink and MaTeX need to do it. Neither are able to do a full cleanup at the moment. Standard packages do it too, e.g. CCompilerDriver`.

Is it really possible to do it? Yes. Compile creates shared libraries when compiling to C code. These do get cleaned up on exit. How is this implemented?

In[1]:= cf = Compile[{{x}}, 2 x, CompilationTarget -> "C"];

In[2]:= FileNames[
 FileNameJoin[{$UserBaseDirectory, "ApplicationData", 
   "CCompilerDriver", "BuildFolder", "*", "*"}]]

Out[2]= {"/Users/szhorvat/Library/Mathematica/ApplicationData/\
CCompilerDriver/BuildFolder/hawkeye-8727/compiledFunction0.dylib"}

In[3]:= Quit

In[1]:= FileNames[
 FileNameJoin[{$UserBaseDirectory, "ApplicationData", 
   "CCompilerDriver", "BuildFolder", "*", "*"}]]

Out[1]= {}

We can use $Epilog for this! $Epilog is by default defined to evaluate << end`. This file does not exist by default (it is not found by FindFile). One possible solution is to create this file and add the cleanup code to it at the time when the package loads. But it is not clear how to this in a robust way, without conflicting with other packages that might also modify the same file.

The cleanup mechanism for shared libraries created by Compile does not seem to use this. Thus I am still hoping to find a better and more robust solution. What does Compile use?

One way to try to find out what happens when existing the kernel is to evaluate On[], followed by Quit[]. This is best done in a terminal session, which does not need to deal with front-end interaction. This trick does not reveal any other evaluations that the one triggered by $Epilog. The default value of $Epilog in Mathematica 10 is If[FindFile["end`"] =!= $Failed, Get["end`"]].

Improve this question
$\endgroup$
9
  • 3
    $\begingroup$ Silly question: what about cleaning at start? Less silly question: isn't there any way to deal with this on OS level, like attaching a 'listener' to the process that will do this when the process is terminated? $\endgroup$
    – Kuba
    Commented Jan 19, 2016 at 10:46
  • $\begingroup$ @Kuba To do it at the OS level is a good idea! It can be a separate process communicating through MathLink, or even simpler: a LibraryLink library which does the cleanup on "uninitialization". I would still like to know how the Compile stuff achieves it though. $\endgroup$
    – Szabolcs
    Commented Jan 19, 2016 at 10:55
  • $\begingroup$ @Kuba I realize I checked the value of $Epilog incorrectly. It does have a value, it will do <<end` if such a file exists. It is not what cleans it up though. What I tried now was to evaluate On[] before exit (in a terminal, no front end!) to see what gets evaluated. I'm trying to sort through that ... $\endgroup$
    – Szabolcs
    Commented Jan 19, 2016 at 11:12
  • $\begingroup$ I am interested in an answer to this, and in particular your solution by including cleanup code in WolframLibrary_uninitialize. Could you post a sample code that cleans up a few known files? $\endgroup$
    – QuantumDot
    Commented Jul 8, 2017 at 5:09
  • $\begingroup$ @QuantumDot I'm still travelling for a week, can you remind me after that please? $\endgroup$
    – Szabolcs
    Commented Jul 12, 2017 at 5:20

1 Answer 1

Reset to default
7
$\begingroup$

I have found $Epilog utterly unreliable for this. Not all ways of quitting Mathematica evaluates $Epilog. And of course, there is the problem that a different package may (re)set it.

I have no idea how Compile implements this, but I have two suggestions:

  1. Place all your temporary files in a special folder that you empty on load. Be careful how you handle parallel kernels that may load multiple versions of your package at the same or interleaving times.

  2. Write a small MathLink program that you Install on startup, has a function AddFileForDeletion that adds the path for a file to an internal list (in the C/C++ world), which it then iterates over and deletes after losing the MathLink connection.

Suggestion 2 is the most reliable way I've found to do temporary file cleanup.

Improve this answer
$\endgroup$
11
  • $\begingroup$ Thanks! I was also thinking of doing something like 2., but instead of a MathLink program it is also possible to use a LibraryLink library (and put the cleanup code in WolframLibrary_uninitialize. Would it at all help if I sent an email to support and suggested to add a reliable builtin cleanup mechanism? $\endgroup$
    – Szabolcs
    Commented Apr 21, 2016 at 7:40
  • 1
    $\begingroup$ I certainly agree, it's ridiculous to have to go to such lengths for something as simple as this. $\endgroup$
    – Malte Lenz
    Commented Apr 21, 2016 at 8:45
  • 5
    $\begingroup$ I think it's unlikely the LibraryLink proposal would work. The FE kills the kernel by sending MLTerminateMessage, which triggers a signal handler, which promptly exits. The advantage of this is that it always works promptly, no matter how hard the kernel might be hung. The disadvantage is that signal handlers are very limiting. You can't even call malloc or free in one, and definitely file system calls are right out. I would be surprised if the signal handler calls LibraryLink uninitialize functions. That having been said, I'm not an expert in the kernel functionality here. $\endgroup$
    – John Fultz
    Commented Jul 9, 2017 at 5:28
  • 5
    $\begingroup$ That having been said, it's not a ridiculous suggestion that the FE ought to try to get the kernel to quit on its own, and only terminate it if it fails to do so after a brief timeout. That would enable end.m, WolframLibrary_uninitialize, etc., to run much more frequently...but there would still be no guarantees. $\endgroup$
    – John Fultz
    Commented Jul 9, 2017 at 5:39
  • 1
    $\begingroup$ @Szabolcs I've not looked at the implementation for LibraryLink on this. I'm surprised by your results (but not surprised that you're running into limitations), though, and I'm very confident about my statements regarding signal handling and MLTerminateMessage. In fact, I recently fixed a bug in WSTP/MathLink for 11.2 which could cause a zombie kernel because MathLink was...ahem...calling malloc inside of a signal handler. Under Linux, the MathLink signal handler was getting invoked sometimes inside of the kernel calling malloc, and it hung hard. $\endgroup$
    – John Fultz
    Commented Jul 31, 2017 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.