4
$\begingroup$

Version: all 12.x

OS: Windows 10 and MacOS

I am having an issue with Export not clearing memory after it is done with exporting to an .mx file. I have reviewed other posts concering memory issues with Export, however they are addressing memory issues while Export is running.

Consider the following:

poly = (x y z + x w z + w y z + x w y)^250;

I am expanding poly using Expand[] and saving the result to polyExpanded.mx file by using Export[]. This file takes up roughly 90MB of disk space (I am actually working with much larger polynomials, this one is for the purposes of a minimal working example).

I am recording MemoryInUse[] at each step: before expanding, after expanding, after exporting the expanded polynomial, and after removing the expanded polynomial from memory using Remove[].

In[1]:= $HistoryLength = 0;

In[2]:= poly = (x y z + x w z + w y z + x w y)^250;

In[3]:= memoryBeforeExpand = MemoryInUse[];
polyExpanded = Expand[poly];
memoryAfterExpand = MemoryInUse[];
Export["polyExpanded.mx", polyExpanded];
memoryAfterExport = MemoryInUse[];
Remove[polyExpanded];
memoryAfterRemove = MemoryInUse[];

The result is as follows:

In[10]:= {memoryBeforeExpand, memoryAfterExpand, memoryAfterExport, 
  memoryAfterRemove}/(1024.)^2

Out[10]= {90.7346, 1225.58, 595.232, 595.231}

Here is what is confusing to me: after expansion 1225MB of memory is occupied, but after Exporting only 595MB is occupied, even though I have done nothing to polyExpanded yet. And now after I remove polyExpanded, only a tiny amount of memory is recovered.

However, if I start a fresh kernel and just comment-out the Export line, I get this:

In[9]:= {memoryBeforeExpand, memoryAfterExpand, memoryAfterExport, 
  memoryAfterRemove}/(1024.)^2

Out[9]= {92.1213, 1110.97, 1110.97, 92.1513}

Hence Remove[] is in fact clearing memory as expected in the absence of Export[]. But as soon as I try and export to file, roughly 500MB are added to memory which I can not clear.

This becomes an issue when I have a collection of polynomials {poly_1,poly_2,...} which I want to expand and save to disk, because Export keeps adding something to memory and doesn't clear it, which eventually results in running out of memory, even though I have sufficient memory to expand each individual poly_i.

I have tried unprotecting and clearing Export, as well as In and Out, but no additional memory was recovered.

I am convinced that this is an issue with Export[] because when I use DumpSave[] in place of Export, I obtain this:

In[1]:= $HistoryLength = 0;

In[2]:= poly = (x y z + x w z + w y z + x w y)^250;

In[3]:= memoryBeforeExpand = MemoryInUse[];
polyExpanded = Expand[poly];
memoryAfterExpand = MemoryInUse[];
DumpSave["polyExpanded.mx", polyExpanded];
memoryAfterDumpSave = MemoryInUse[];
Remove[polyExpanded];
memoryAfterRemove = MemoryInUse[];

In[10]:= {memoryBeforeExpand, memoryAfterExpand, memoryAfterDumpSave, 
  memoryAfterRemove}/(1024.)^2

Out[10]= {89.9266, 1108.77, 323.722, 89.9566}

Here Remove[] clears memory as expected, and at the end approx. the same amount of memory is in use as it was before expansion took place.

However I am still confused as to why less memory is occupied after expanding and saving than after just expanding. I would have expected to have approx. the same amount of memory in use after expanding, and after expanding and saving since the saving process terminated succesfully but I have not yet removed polyExpanded from memory.

Questions:

Is it possible to clear the memory that Export is not releasing? Is DumpSave a safe work-around, i.e. is the data saved by Export and DumpSave equivalent when saving as .mx?

$\endgroup$
4
  • 1
    $\begingroup$ DumpSave is not a workaround in this case, but the correct way to save data to an MX file. Export is most certainly calling DumpSave internally. Also, does the problem go away after calling ClearAll[System`Private`ConvertersPrivateDumpSymbol]? $\endgroup$
    – Jason B.
    Oct 25, 2021 at 11:25
  • $\begingroup$ Thank you for you reply @JasonB. Your suggestion does indeed resolve the problem, and memory is almost fully recovered (a few kB are not recovered which is expected). If you write that as an answer, I will accept. $\endgroup$ Oct 25, 2021 at 19:01
  • $\begingroup$ This seems to be fixed in Mathematica 13.0 $\endgroup$
    – rhermans
    Oct 28, 2021 at 10:52
  • $\begingroup$ @rhermans Mathematica 13.0? Is there already a list of features/bug fixes somewhere (I wasn't able to google it myself)? $\endgroup$ Oct 28, 2021 at 17:52

1 Answer 1

5
$\begingroup$

As explained in another post, Export uses DumpSave under the hood for the MX format. It does this by setting an internal private symbol equal to the data to be exported, and then dumps that symbol to the MX file. This works great, but not clearing the symbol afterwards is a memory leak.

As a workaround you can use these functions is to call ClearAll[System`Private`ConvertersPrivateDumpSymbol] after any MX importing or exporting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.