14
$\begingroup$

Bug introduced in 10.0 and persisting though 11.1.1


This problem that I would like to resolve is how to free memory used Import a file when the file is no longer in use.

I have a project in which I need to read and analyze a large number of .json files. Each file can be as large as 40MB. The files are used in code roughly as follows:

Table[analysis[Import[ filePaths[[p]] ]], {p,1,Length[filePaths]}]

where analysis is a function that includes no side effects, and filePaths is a list of strings giving paths to files.

Before and after the analysis the values reported MemoryInUse[] and MemoryInUse[$FrontEnd] are essentially unchanged, but I can see that the memory consumed by the WolframKernel has grown as each file is opened.

I assume that the entire imported file is somehow lingering in memory. Finally, if I try to re-import the file I will get errors, which suggests some kind of lingering hold on the file.

$\endgroup$
  • $\begingroup$ How exactly are you measuring memory use? Do I understand it correctly that MemoryInUse[] and some other method (task manager?) don't agree about whether memory usage is growing? $\endgroup$ – Szabolcs Aug 4 '15 at 9:50
  • $\begingroup$ Thank you leosenko and Szabolcs for taking the time to verify the problem. (Especially for going through earlier versions!) Szabolcs you said that "since version 10.0 the JSON importer uses C functions through LibraryLink" - is there a way to work around this? For example, is there another method to import files, or a way to specify that a different library is used? $\endgroup$ – AngelGabriel Aug 5 '15 at 5:37
  • $\begingroup$ Szabolcs - the memory usage that you demonstrated is what I have seen as well, measured with both MemoryInUse and with OS X Activity Monitor. My goal is to automate the analysis of a large number of files, and this memory issue is the remaining obstacle. $\endgroup$ – AngelGabriel Aug 5 '15 at 9:23
  • $\begingroup$ WORK-AROUND: Launching and killing kernels successfully keeps the memory leak in check. See: mathematica.stackexchange.com/questions/91317/… $\endgroup$ – AngelGabriel Aug 12 '15 at 5:26
13
$\begingroup$

tl;dr I think it's a memory leak (bug) and you should report it to Wolfram Support (please do!)


According to my reading, you were saying that after importing JSON files many times, the kernel memory usage reported by the operating system (or some task manager program) was growing to unreasonable levels. However, the memory usage reported by the kernel itself through MemoryInUse[] didn't increase.

So I tested this using the following in a fresh kernel:

In[1]:= $HistoryLength = 0;

In[2]:= MemoryInUse[]

Out[2]= 28021176

In[5]:= Monitor[
 Do[json = 
    Import["somefile.json", 
     "JSON"];, {i, 1000}],
 i
 ]

Out[5]= $Aborted

In[7]:= MemoryInUse[]    
Out[7]= 32725480

Indeed the MemoryInUse[] value does not increase, but the true memory usage of the kernel, as reported by Activity Monitor on OS X, kept growing. After ~500 iterations it exceeded 3 GB.

enter image description here

To me this looks like a memory leak, i.e. a bug.


So what could be the reason for MemoryInUse[] not showing this excessive memory usage?

I think it's because since version 10.0 the JSON importer uses C functions through LibraryLink (one can check this with a bit of spelunking), and memory allocated through these is not tracked by MemoryInUse[]. Indeed I can reproduce the problem in all of versions 10.0.2, 10.1.0 and 10.2.0, but not in 9.0.1. In version 9 and earlier the JSON importer used a different Java based implementation. In v9 the kernel memory usage stays around 60-70 MB with a JavaAplicationStub process staying around 125 MB, no matter how many times I import a JSON file.

enter image description here

$\endgroup$
  • $\begingroup$ +1 for this analysis, bug report submitted. $\endgroup$ – ilian Aug 4 '15 at 17:41
8
$\begingroup$

So if you have:

file = Import[path]

then after the file is not needed:

Clear[file]

e.g.:

In[77]:= MemoryInUse[]
file = Import[StringJoin[NotebookDirectory[], "IMG_3025.jpg"]];
MemoryInUse[]
Clear[file];
MemoryInUse[]

Out[77]= 79593488

Out[79]= 115619864

Out[81]= 79591456
$\endgroup$
  • $\begingroup$ Could you show the impact of Clear[file] on the memory usage? $\endgroup$ – Dr. belisarius Aug 3 '15 at 23:06
  • $\begingroup$ @belisarius, I added an example. $\endgroup$ – leosenko Aug 4 '15 at 1:24
  • 1
    $\begingroup$ However, this is not the reason for the extreme memory usage the OP sees. He said that the value reported by MemoryInUse[] doesn't even increase. $\endgroup$ – Szabolcs Aug 4 '15 at 18:14
3
$\begingroup$

I am having this same issue with Mathematica 11.0.1 under Windows 7. I have this minimal example:

Scan[ImportString["{\"field\":\"value\"}\n", "JSON"] &, Range[100000]]

While MemoryInUse[] remains constant, the kernel keeps using more and more memory as reported by Windows Process Explorer.

This second example, with a much larger JSON document results in a much faster memory leak:

longText = FromCharacterCode@RandomInteger[{65, 90}, 1000];

Scan[ImportString["{\"field\":\"" <> longText <> "\"}\n", "JSON"]; &, Range[100000]]

I have reported this to Wolfram support and will update this when I get a response.

UPDATE: I just found that this bug can be avoided by using "RawJSON" instead of "JSON". And it is also faster.

UPDATE 2: This bug was confirmed by Wolfram support (CASE:3855578).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.