Using PredictorMeasurements with a neural net?

Question

PredictorMeasurements doesn't work with NetGraph, here's an example:

makeRule[a_, b_] := 
  IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}];
enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];
net = NetInitialize@
   NetChain[{UnitVectorLayer[], LongShortTermMemoryLayer[40], 
     LongShortTermMemoryLayer[20], SequenceLastLayer[], 
     LinearLayer[]}, "Input" -> enc, "Output" -> "Real"];
PredictorMeasurements[net, data, "Accuracy"]

Is there any way to make this work? Perhaps converting the net into a predictor?

Answer 1

Let's look under the hood of Predict.

p = Predict[{{1, 2} -> 3, {2, 3} -> 4}, 
   Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"}];

Options[p][[1]]["Model"]["Network"]

The network has 2 outputs: mean and log-variance.

Options[p][[1]]["Model"]["Options"]["Network"]["Value"]

Loss function is very interesting:

Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"]

And now let's replace trained network in Predict with our custom net.

net = NetGraph[
   {
    LongShortTermMemoryLayer[40],
    NetMapOperator[LinearLayer[10]],
    LongShortTermMemoryLayer[20],
    SequenceLastLayer[],
    LinearLayer[100],
    Ramp,
    LinearLayer[2],
    PartLayer[1 ;; 1],
    PartLayer[2 ;; 2]
    },
   {1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> {8, 9}, 
    8 -> NetPort["logvariance"], 9 -> NetPort["mean"]},
   "Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
   ] // NetInitialize

GeneralUtilities`PrintDefinitions@PredictorFunction

We can see that PredictorFunction expects Association as the input.

assoc = Options[p][[1]];
assoc["Model"]["Network"] = net;
p1 = PredictorFunction[assoc]

We can make predictions:

p1[{{1, 2}, {2, 3}}]

{3.49756, 3.50435}

And we can do PredictorMeasurements:

pm1 = PredictorMeasurements[p1, {{1, 2} -> 3, {2, 3} -> 4}]

pm1["MeanSquare"]

0.246618

Addendum

makeRule[a_, b_] := IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}] // Flatten;

enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];

fe = FeatureExtraction[data[[;; , 1]], enc];

p = Predict[
   data[[-2 ;;, 1]] -> data[[-2 ;;, 2]],
   Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"},
   FeatureExtractor -> fe
   ];

net = NetGraph[
  {
   (* UnitVectorLayer does not supported because of Standardize as the data processor *)
   LongShortTermMemoryLayer[40],
   LongShortTermMemoryLayer[20],
   SequenceLastLayer[],
   LinearLayer[2],
   PartLayer[1 ;; 1],
   PartLayer[2 ;; 2]
   },
  {1 -> 2 -> 3 -> 4 -> {5, 6}, 5 -> NetPort["logvariance"], 6 -> NetPort["mean"]}, 
  "Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
  ];

loss = Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"];

net = NetGraph[
   {
    net,
    loss
    },
   {
    NetPort["Input"] -> 1,
    NetPort[1, "logvariance"] -> NetPort[2, "Input1"],
    NetPort[1, "mean"] -> NetPort[2, "Input2"],
    NetPort["Target"] -> NetPort[2, "Target"]
    }
   ];

netT = NetTrain[
   net,
   <|
    "Input" -> (Partition[#, 1] & /@ Standardize /@ enc@data[[;; , 1]]),
    "Target" -> Partition[data[[;; , 2]], 1],
    "Output" -> data[[;; , 2]]
    |>,
   MaxTrainingRounds -> 1
   ];

netT = NetExtract[netT, 1];

assoc = Options[p][[1]];
assoc["Model"]["Network"] = netT;
p1 = PredictorFunction[assoc];

pm1 = PredictorMeasurements[p1, data]