Export is not clearing memory

Question

Version: all 12.x

OS: Windows 10 and MacOS

I am having an issue with Export not clearing memory after it is done with exporting to an .mx file. I have reviewed other posts concering memory issues with Export, however they are addressing memory issues while Export is running.

Consider the following:

poly = (x y z + x w z + w y z + x w y)^250;

I am expanding poly using Expand[] and saving the result to polyExpanded.mx file by using Export[]. This file takes up roughly 90MB of disk space (I am actually working with much larger polynomials, this one is for the purposes of a minimal working example).

I am recording MemoryInUse[] at each step: before expanding, after expanding, after exporting the expanded polynomial, and after removing the expanded polynomial from memory using Remove[].

In[1]:= $HistoryLength = 0;

In[2]:= poly = (x y z + x w z + w y z + x w y)^250;

In[3]:= memoryBeforeExpand = MemoryInUse[];
polyExpanded = Expand[poly];
memoryAfterExpand = MemoryInUse[];
Export["polyExpanded.mx", polyExpanded];
memoryAfterExport = MemoryInUse[];
Remove[polyExpanded];
memoryAfterRemove = MemoryInUse[];

The result is as follows:

In[10]:= {memoryBeforeExpand, memoryAfterExpand, memoryAfterExport, 
  memoryAfterRemove}/(1024.)^2

Out[10]= {90.7346, 1225.58, 595.232, 595.231}

Here is what is confusing to me: after expansion 1225MB of memory is occupied, but after Exporting only 595MB is occupied, even though I have done nothing to polyExpanded yet. And now after I remove polyExpanded, only a tiny amount of memory is recovered.

However, if I start a fresh kernel and just comment-out the Export line, I get this:

In[9]:= {memoryBeforeExpand, memoryAfterExpand, memoryAfterExport, 
  memoryAfterRemove}/(1024.)^2

Out[9]= {92.1213, 1110.97, 1110.97, 92.1513}

Hence Remove[] is in fact clearing memory as expected in the absence of Export[]. But as soon as I try and export to file, roughly 500MB are added to memory which I can not clear.

This becomes an issue when I have a collection of polynomials {poly_1,poly_2,...} which I want to expand and save to disk, because Export keeps adding something to memory and doesn't clear it, which eventually results in running out of memory, even though I have sufficient memory to expand each individual poly_i.

I have tried unprotecting and clearing Export, as well as In and Out, but no additional memory was recovered.

I am convinced that this is an issue with Export[] because when I use DumpSave[] in place of Export, I obtain this:

In[1]:= $HistoryLength = 0;

In[2]:= poly = (x y z + x w z + w y z + x w y)^250;

In[3]:= memoryBeforeExpand = MemoryInUse[];
polyExpanded = Expand[poly];
memoryAfterExpand = MemoryInUse[];
DumpSave["polyExpanded.mx", polyExpanded];
memoryAfterDumpSave = MemoryInUse[];
Remove[polyExpanded];
memoryAfterRemove = MemoryInUse[];

In[10]:= {memoryBeforeExpand, memoryAfterExpand, memoryAfterDumpSave, 
  memoryAfterRemove}/(1024.)^2

Out[10]= {89.9266, 1108.77, 323.722, 89.9566}

Here Remove[] clears memory as expected, and at the end approx. the same amount of memory is in use as it was before expansion took place.

However I am still confused as to why less memory is occupied after expanding and saving than after just expanding. I would have expected to have approx. the same amount of memory in use after expanding, and after expanding and saving since the saving process terminated succesfully but I have not yet removed polyExpanded from memory.

Questions:

Is it possible to clear the memory that Export is not releasing? Is DumpSave a safe work-around, i.e. is the data saved by Export and DumpSave equivalent when saving as .mx?

Answer 1

As explained in another post, Export uses DumpSave under the hood for the MX format. It does this by setting an internal private symbol equal to the data to be exported, and then dumps that symbol to the MX file. This works great, but not clearing the symbol afterwards is a memory leak.

As a workaround you can use these functions is to call ClearAll[System`Private`ConvertersPrivateDumpSymbol] after any MX importing or exporting.