13
$\begingroup$
$HistoryLength = 0;

Warming up:

TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]];

No memory leak:

MemoryInUse[]

118502448

Do[
 StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100],
 {i, 10000}
 ]
MemoryInUse[]

118502448

Memory leak in TextSentences:

Do[
 TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]],
 {i, 10000}
 ] // AbsoluteTiming

{32.6916,Null}

MemoryInUse[]

260521400

How to free memory without restarting the kernel?

Possible workaround (but very slow - up to 5x):

LaunchKernels[1];

MemoryInUse[]

263313480

Do[
  ParallelEvaluate@TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]],
  {i, 10000}
  ] // AbsoluteTiming

{161.063, Null}

MemoryInUse[]

263313480

$\endgroup$
  • $\begingroup$ Check the help files for Share $\endgroup$ – Bill Watts Jul 12 at 17:41
  • $\begingroup$ @BillWatts Thank you for your comment! Share will not help. Here is the good answer. But I am looking for a workaround without parallel kernel. $\endgroup$ – Alexey Golyshev Jul 12 at 18:58
13
+50
$\begingroup$

Please report this to support. You can get rid of some of the memory leak by executing the following:

cleanupTextSentences[] := Remove[
    "NaturalLanguageProcessing`TextCases`PackagePrivate`interpretations$@",
    "NaturalLanguageProcessing`TextCases`PackagePrivate`probabilities$@",
    "NaturalLanguageProcessing`TextPosition`PackagePrivate`punctuationPositions$@",
    "NaturalLanguageProcessing`TextPosition`PackagePrivate`tokenPositions$@",
    "NaturalLanguageProcessing`TextPosition`PackagePrivate`wordPositions$@"
]

Your example:

MemoryInUse[]

Do[
    TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]],
    {i, 10000}
] // AbsoluteTiming

MemoryInUse[]

cleanupTextSentences[]
MemoryInUse[]

123158784

{26.6266, Null}

231403704

167386720

If you want to use parallel processing, why not use ParallelDo?

MemoryInUse[]

ParallelDo[
    TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"],{" ","."}],100]],
    {i,10000}
] //AbsoluteTiming

MemoryInUse[]

182588776

{7.46125, Null}

182591464

$\endgroup$
  • $\begingroup$ Thank you for your answer! Already reported. I will process my data sequentially. Do you know a workaround for ANY system function that may have a memory leak? $\endgroup$ – Alexey Golyshev Jul 16 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.