Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am calling lots of OpenCL functions from Mathematica. The problem is that the memory the kernel uses grows and grows until the RAM is fully occupied.

I have created a small piece of code to demonstrate the behaviour.

Needs["OpenCLLink`"];
openclPlatform = 1; openclDevice = 1;

For[i = 0, i < 1000, i++,
 OpenCLMersenneTwister[1, Platform -> openclPlatform, 
   Device -> openclDevice]; 
 ]

(OpenCLMersenneTwister is a predefined function by Mathematica)

When running that piece of code multiple times I can see that the RAM usage of MathKernel.exe grows and no memory is freed.

Is it a bug in Mathematica or is my code wrong?

Thanks

share|improve this question

1 Answer 1

up vote 3 down vote accepted

There is a similar bug in CUDALink. It appears to be related to memory management of MTensor objects passed back to Mathematica from the CUDALink or OpenCLLink library. The problem, as far as I can tell, arises due to the LibraryFunction defined in OpenCLLink`Private`cGetMemory["Double"] which specifies a shared return. This means that the library will keep ownership of the returned MTensor until it is explicitly disowned. Presumably there is a bug in the library code so that it never disowns the MTensor and the memory is never freed.

My workaround is to unload the cGetMemory function and then reload it, specifying automatic return instead of shared return. This should cause the library to lose ownership of the MTensor when it is returned to Mathematica, so the memory will be freed as soon as Mathematica has done with it.

Example:

Needs["OpenCLLink`"];

(* kernel memory grows by size of returned array on every call *)
Table[OpenCLMersenneTwister[100000]; MemoryInUse[], {5}] // Differences
(* {820520, 819352, 819352, 819352} *)

(* apply the fix *)
With[{func := OpenCLLink`Private`cGetMemory["Double"]},
  Module[{args = List @@ func},
   LibraryFunctionUnload[func];
   func = LibraryFunctionLoad @@ Delete[args, {4, 3}]]];

(* kernel memory is now under control *)
Table[OpenCLMersenneTwister[100000]; MemoryInUse[], {5}] // Differences
(* {16, 16, 16, 16} *)

If your GPU is single precision only you probably need to use "Single" instead of "Double". To similarly patch CUDALink just put CUDALink in place of OpenCLLink.

I can't be certain that there aren't any nasty side effects to this workaround, as I don't understand LibraryLink very well, but I have used this fix on CUDALink and been able to run code for >24 hours, transferring tens of gigabytes from the GPU back to host memory without the Mathematica kernel running out of memory.

share|improve this answer
    
This fixes the memory issue. Thank you very much! Without you we would have never found this fix. –  user2224780 Mar 19 at 17:12
    
Maybe it really has side effects. I now get "OpenCLFunction::outmem: OpenCLLink ran out of available memory, possibly due to not freeing memory using the memory manager.". But all memory is freed using OpenCLMemoryUnload –  user2224780 Mar 20 at 11:45
    
@user2224780, that's a shame, as I said it has been working for me with CUDALink. I'm not sure if that message refers to host or device memory - what does the mathkernel memory usage look like when you get that error? –  Simon Woods Mar 20 at 13:29
    
MathKernel.exe looks normal and does not grow. It must be device memory... –  user2224780 Mar 20 at 13:32
    
@user2224780, if you're sure you are freeing all the memory you've allocated then either my fix is breaking something as you suggested, or there is another bug in OpenCLLink. Sorry I can't be more help! Maybe Wolfram technical support can help... –  Simon Woods Mar 20 at 14:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.