4
$\begingroup$

When I train a neural network with NetTrain[..., TargetDevice->"GPU"], wolfram's neural network code automatically allocates memory on my GPU for my model's computation graph, copied depending on the batch size.

My problem is that if I change my model and re-run NetTrain, memory allocated for the old model is not reliably de-allocated from my gpu, resulting in an error:

NetTrain: An unknown internal error occurred. Consult internal`$LastInternalFailure for potential information.

and internal`$LastInternalFailure of:

MXNetError <...> ./pooled_storage_manager.h:161: cudaMalloc retry failed: out of memory

How can I explicitly tell the wolfram neural network library to clear out allocated GPU memory?

(I have found a workaround without restarting my computer involving repeating restarting the Wolfram Kernel and waiting dozens of seconds until my gpu memory is freed. But this is extremely inconvenient, and it requires me to re-run my notebook to continue from where I left off.)

$\endgroup$
5
  • 1
    $\begingroup$ Please report this to Wolfram technical support. $\endgroup$
    – Jason B.
    Aug 14, 2021 at 13:10
  • 2
    $\begingroup$ Any update on this from Wolfram? $\endgroup$
    – Edmund
    Apr 10, 2022 at 17:55
  • $\begingroup$ I do not know; I have not tested this in 13.1 I did reach out to support when I made this issue, but they basically told me to kill the kernel in the Windows task manager :( $\endgroup$ Apr 16, 2022 at 5:52
  • 1
    $\begingroup$ No this is still not fixed. I still get the same error. $\endgroup$ May 9, 2022 at 8:04
  • $\begingroup$ I meant 13.0.1, 13.1 (released in 2022-06-29) seems to be fixed!! I'm writing an answer now. $\endgroup$ Jul 31, 2022 at 19:49

1 Answer 1

3
$\begingroup$

This issue appears to be resolved in Wolfram 13.1! (the memory is not freed by mxnet, but at least I can ClearAll the model and re-train and without the GPU memory consumption spiking).

Repeatedly running the following cell in a notebook does not result in GPU memory allocation errors anymore (note, this uses about 4 gb of VRAM for me, but you could reduce the number of layers if you want to test on a GPU with less VRAM).

$HistoryLength = 0;
ClearAll[data, trained, net]
data = Flatten@
   Table[{x, y} -> Exp[-Norm[{x, y}]], {x, -3, 3, .005}, {y, -3, 
     3, .005}];
net = NetChain[{300, Tanh, 3000, Sin, 3000, Sin, 3000, Sin, 3000, 
    Sin, 3000, Tanh, 1}];
trained = 
 NetTrain[net, data, BatchSize -> 1024, TargetDevice -> "GPU"]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.