5
$\begingroup$

I am trying to train a UNet on my laptop (Lenovo P72) and I obtain the following error when I use NetTrain on it:

CompiledFunction::cfta: Argument {Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]} at position 1 should be a rank 1 tensor of machine-size real numbers.

And when I observe the trained network, some parts are uninitialized (in red) and others are initialized (blue/green/purple).

Someone has an idea?

A point which is very strange, when I set TargetDevice->"GPU" with a learning set of 1 input/output, it works correctly. When I use the CPU instead, with a set of inputs/outputs as big as I want, it is OK too. But when I set TargetDevice ->"GPU" and that I have a number of inputs/outputs greater than one, I still obtain this error.

This occurs on Mathematica 12.0.0.0.

Thanks for your help.

$\endgroup$
  • $\begingroup$ Oh I forgot to say that there is another trick: when I set BatchSize->1 (with GPU support), it still works, whatever the number of inputs/outputs, but obviously it's very slow ... and my goal is to be able to reach a batchsize of 8 at least. $\endgroup$ – Nicolas Boutry May 6 at 0:52
  • $\begingroup$ It seems that the problem is due to the size of needed VRAM to train the network: when I sufficiently increase the batchsize or the number of nodes in the neural net, I observe that the VRAM is full during the training and I obtain the same kind of error. If my hypothesis is true, it means that with my 16GB Nvidia Quadro, I won't be able to train my UNet with a BatchSize bigger than 1. However, when I use the CPU, then the used memory is the "usual" memory, and then I do not get the same limitations. $\endgroup$ – Nicolas Boutry May 7 at 4:28
  • $\begingroup$ I get exactly the same errors it works fine training datasets with small data (e.g. 2D 128x128) but when i move to bigger datasets (e.g. 3D 40x128x128) it starts to also give these errors. Im using a Titan XP. ![enter image description here](i.stack.imgur.com/4TJsh.png) Weirdly the training seems to run regardless. The same functions and training work fine in Mathematica 11. $\endgroup$ – Martijn Froeling May 24 at 12:55
  • $\begingroup$ Seems like a bug (though I can't confirm) - consider reporting it to WRI. $\endgroup$ – Carl Lange May 24 at 13:10
  • $\begingroup$ where's your UNet model and codes $\endgroup$ – HyperGroups May 24 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.