# Using PredictorMeasurements with a neural net?

PredictorMeasurements doesn't work with NetGraph, here's an example:

makeRule[a_, b_] :=
IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}];
enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];
net = NetInitialize@
NetChain[{UnitVectorLayer[], LongShortTermMemoryLayer[40],
LongShortTermMemoryLayer[20], SequenceLastLayer[],
LinearLayer[]}, "Input" -> enc, "Output" -> "Real"];
PredictorMeasurements[net, data, "Accuracy"]


Is there any way to make this work? Perhaps converting the net into a predictor?

Let's look under the hood of Predict.

p = Predict[{{1, 2} -> 3, {2, 3} -> 4},
Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"}];

Options[p][[1]]["Model"]["Network"]


The network has 2 outputs: mean and log-variance.

Options[p][[1]]["Model"]["Options"]["Network"]["Value"]


Loss function is very interesting:

Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"]


And now let's replace trained network in Predict with our custom net.

net = NetGraph[
{
LongShortTermMemoryLayer[40],
NetMapOperator[LinearLayer[10]],
LongShortTermMemoryLayer[20],
SequenceLastLayer[],
LinearLayer[100],
Ramp,
LinearLayer[2],
PartLayer[1 ;; 1],
PartLayer[2 ;; 2]
},
{1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> {8, 9},
8 -> NetPort["logvariance"], 9 -> NetPort["mean"]},
"Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
] // NetInitialize


GeneralUtilitiesPrintDefinitions@PredictorFunction


We can see that PredictorFunction expects Association as the input.

assoc = Options[p][[1]];
assoc["Model"]["Network"] = net;
p1 = PredictorFunction[assoc]


We can make predictions:

p1[{{1, 2}, {2, 3}}]


{3.49756, 3.50435}

And we can do PredictorMeasurements:

pm1 = PredictorMeasurements[p1, {{1, 2} -> 3, {2, 3} -> 4}]


pm1["MeanSquare"]


0.246618

makeRule[a_, b_] := IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}] // Flatten;

enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];

fe = FeatureExtraction[data[[;; , 1]], enc];

p = Predict[
data[[-2 ;;, 1]] -> data[[-2 ;;, 2]],
Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"},
FeatureExtractor -> fe
];

net = NetGraph[
{
(* UnitVectorLayer does not supported because of Standardize as the data processor *)
LongShortTermMemoryLayer[40],
LongShortTermMemoryLayer[20],
SequenceLastLayer[],
LinearLayer[2],
PartLayer[1 ;; 1],
PartLayer[2 ;; 2]
},
{1 -> 2 -> 3 -> 4 -> {5, 6}, 5 -> NetPort["logvariance"], 6 -> NetPort["mean"]},
"Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
];

loss = Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"];

net = NetGraph[
{
net,
loss
},
{
NetPort["Input"] -> 1,
NetPort[1, "logvariance"] -> NetPort[2, "Input1"],
NetPort[1, "mean"] -> NetPort[2, "Input2"],
NetPort["Target"] -> NetPort[2, "Target"]
}
];

netT = NetTrain[
net,
<|
"Input" -> (Partition[#, 1] & /@ Standardize /@ enc@data[[;; , 1]]),
"Target" -> Partition[data[[;; , 2]], 1],
"Output" -> data[[;; , 2]]
|>,
MaxTrainingRounds -> 1
];

netT = NetExtract[netT, 1];

assoc = Options[p][[1]];
assoc["Model"]["Network"] = netT;
p1 = PredictorFunction[assoc];

pm1 = PredictorMeasurements[p1, data]

• This is great, but what about for my example in question (RNN on text)? I can't get it to work this way... thanks Mar 19, 2018 at 13:20
• @user5601 I was trying to make as simple example as possible. Answer to your question in addendum. Mar 20, 2018 at 8:39
• Thanks for addressing my comment. I'm still unclear on what "UnitVectorLayer does not supported because of Standardize as the data processor" means? Mar 20, 2018 at 14:58
• Also what do the logvariance and mean NetPorts actually compute themselves, and why do we need to give the output to NetTrain? and must we use the loss NetGraph given by Predict, can't we define this arbitrarily? Mar 20, 2018 at 15:06
• 1. I found a way how UnitVectorLayer` can be used. We can remove some data processors. But there is very strange error. 1drv.ms/u/s!AvVas0AkeAWi0GgEpOWYv9HPswtf Mar 20, 2018 at 19:21