14
$\begingroup$
$HistoryLength = 0;

Warming up:

TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]];

No memory leak:

MemoryInUse[]

118502448

Do[
 StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100],
 {i, 10000}
 ]
MemoryInUse[]

118502448

Memory leak in TextSentences:

Do[
 TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]],
 {i, 10000}
 ] // AbsoluteTiming

{32.6916,Null}

MemoryInUse[]

260521400

How to free memory without restarting the kernel?

Possible workaround (but very slow - up to 5x):

LaunchKernels[1];

MemoryInUse[]

263313480

Do[
  ParallelEvaluate@TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]],
  {i, 10000}
  ] // AbsoluteTiming

{161.063, Null}

MemoryInUse[]

263313480

$\endgroup$
2
  • $\begingroup$ Check the help files for Share $\endgroup$
    – Bill Watts
    Jul 12, 2019 at 17:41
  • $\begingroup$ @BillWatts Thank you for your comment! Share will not help. Here is the good answer. But I am looking for a workaround without parallel kernel. $\endgroup$ Jul 12, 2019 at 18:58

1 Answer 1

13
+50
$\begingroup$

Please report this to support. You can get rid of some of the memory leak by executing the following:

cleanupTextSentences[] := Remove[
    "NaturalLanguageProcessing`TextCases`PackagePrivate`interpretations$@",
    "NaturalLanguageProcessing`TextCases`PackagePrivate`probabilities$@",
    "NaturalLanguageProcessing`TextPosition`PackagePrivate`punctuationPositions$@",
    "NaturalLanguageProcessing`TextPosition`PackagePrivate`tokenPositions$@",
    "NaturalLanguageProcessing`TextPosition`PackagePrivate`wordPositions$@"
]

Your example:

MemoryInUse[]

Do[
    TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"], {" ", "."}], 100]],
    {i, 10000}
] // AbsoluteTiming

MemoryInUse[]

cleanupTextSentences[]
MemoryInUse[]

123158784

{26.6266, Null}

231403704

167386720

If you want to use parallel processing, why not use ParallelDo?

MemoryInUse[]

ParallelDo[
    TextSentences[StringJoin@RandomChoice[Join[Alphabet["Russian"],{" ","."}],100]],
    {i,10000}
] //AbsoluteTiming

MemoryInUse[]

182588776

{7.46125, Null}

182591464

$\endgroup$
1
  • $\begingroup$ Thank you for your answer! Already reported. I will process my data sequentially. Do you know a workaround for ANY system function that may have a memory leak? $\endgroup$ Jul 16, 2019 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.