Version 12.3.1 Problems with NetTrain

Question

Until now I am using MMA 12.2 because NetTrain istn't running stable with 12.3. Yesterday I have downloaded and installed 12.3.1 to check if the problems using NetTrain are gone. I have loaded my Notebook with several Net-Functions. First time I have processed NetTrain, MMA downloads new data from the server (was wondering, anything changed from 12.3 to 12.3.1?). After the download has finished, if got the Message "TargetDevice -> "GPU" requires a restart of your Wolfram Language session" Ok, I have closed Mathermatica and checked (using ProcessExplorer) if there is running anything in the background. I have restarted MMA, now NetTrain didn't load new data from the Wolfram-Server, but I get the same message: "TargetDevice -> "GPU" requires a restart of your Wolfram Language session" Strange, ok, I have restartet the computer, tried it again, and again, I get the same message. For me it looks like the MMA 12.3.1 has a completely broken Net-Functionality. Are there any ideas how to fix it?

Answer 1

This was caused by an improper paclet update. The MXNetResources paclet with version 12.4.407 should not be used by Mathematica 12.3.1. We have removed it from our servers and apologize for the disruption.

We will be pushing another paclet update that will automatically delete the troublesome update, but in the meantime those affected can restore the proper behavior by doing the following:

PacletUninstall["MXNetResources" -> "12.4.407"]

Answer 2

This is happening because an MXNetResources paclet intended for Wolfram internal use only was made available to users of 12.3/12.3.1 by mistake. We are very sorry for this.

We have fixed the paclets on our server so that users will not be exposed to this problem for now on. We are also preparing another update to fix the setup of users who encountered the problem and are now stuck with a bad configuration. It should be out early next week.

In the meantine, the problem can be fixed manually by uninstalling the unintended MXNetResources update: PacletUninstall["MXNetResources" -> "12.4.407"]. Then Quit and try to use the GPU again: a new download (of the correct paclet) should start, and after another restart everything should go smooth.

Answer 3

Exploration

mxnet.dll has been modified. There are no changes in the rest of the files.

PacletFindRemote["MXNetResources"]

PacletInstall[%[[2]]]

MapThread[
 If[FileHash[#1, "SHA256", All] != FileHash[#2, "SHA256", All],
   FileNameSplit[#1][[-1]],
   Nothing
   ] &
 ,
 {
  FileNames["*", "MXNetResources-WIN64-12.2.404\\LibraryResources\\Windows-x86-64"],
  FileNames["*", "MXNetResources-WIN64-12.4.407\\LibraryResources\\Windows-x86-64"]
  }
 ]

{"mxnet.dll"}

GeneralUtilities`PrintDefinitions@NetTrain

$DefaultContext = ParseTargetDevice @ targetDevice;

Click on ParseTargetDevice.

Then click on $GPUCount.

$GPUCount := With[
    {count = Quiet[Replace[mxlMXGetGPUCount[], Except[_Integer] -> 0]]},
    If[TrueQ[Greater[count, 0]], $GPUCount = count];
    count
];

mxlMXGetGPUCount = LibraryFunction[
    "C:\\Mathematica\\SystemFiles\\Components\\MXNetLink\\LibraryResources\\Windows-x86-64\\MXNetLink.dll",
    "mxlMXGetGPUCount", {}, Integer
];

Needs["MXNetLink`"]

MXNetLink`GPU`PackagePrivate`mxlMXGetGPUCount[]

0