# Using PredictorMeasurements with a neural net?

PredictorMeasurements doesn't work with NetGraph, here's an example:

makeRule[a_, b_] :=
IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}];
enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];
net = NetInitialize@
NetChain[{UnitVectorLayer[], LongShortTermMemoryLayer[40],
LongShortTermMemoryLayer[20], SequenceLastLayer[],
LinearLayer[]}, "Input" -> enc, "Output" -> "Real"];
PredictorMeasurements[net, data, "Accuracy"]

Is there any way to make this work? Perhaps converting the net into a predictor?

Let's look under the hood of Predict.

p = Predict[{{1, 2} -> 3, {2, 3} -> 4},
Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"}];

Options[p][[1]]["Model"]["Network"]

The network has 2 outputs: mean and log-variance.

Options[p][[1]]["Model"]["Options"]["Network"]["Value"]

Loss function is very interesting:

Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"]

And now let's replace trained network in Predict with our custom net.

net = NetGraph[
{
LongShortTermMemoryLayer[40],
NetMapOperator[LinearLayer[10]],
LongShortTermMemoryLayer[20],
SequenceLastLayer[],
LinearLayer[100],
Ramp,
LinearLayer[2],
PartLayer[1 ;; 1],
PartLayer[2 ;; 2]
},
{1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> {8, 9},
8 -> NetPort["logvariance"], 9 -> NetPort["mean"]},
"Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
] // NetInitialize

GeneralUtilities`PrintDefinitions@PredictorFunction

We can see that PredictorFunction expects Association as the input.

assoc = Options[p][[1]];
assoc["Model"]["Network"] = net;
p1 = PredictorFunction[assoc]

We can make predictions:

p1[{{1, 2}, {2, 3}}]

{3.49756, 3.50435}

And we can do PredictorMeasurements:

pm1 = PredictorMeasurements[p1, {{1, 2} -> 3, {2, 3} -> 4}]

pm1["MeanSquare"]

0.246618

makeRule[a_, b_] := IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}] // Flatten;

enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];

fe = FeatureExtraction[data[[;; , 1]], enc];

p = Predict[
data[[-2 ;;, 1]] -> data[[-2 ;;, 2]],
Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"},
FeatureExtractor -> fe
];

net = NetGraph[
{
(* UnitVectorLayer does not supported because of Standardize as the data processor *)
LongShortTermMemoryLayer[40],
LongShortTermMemoryLayer[20],
SequenceLastLayer[],
LinearLayer[2],
PartLayer[1 ;; 1],
PartLayer[2 ;; 2]
},
{1 -> 2 -> 3 -> 4 -> {5, 6}, 5 -> NetPort["logvariance"], 6 -> NetPort["mean"]},
"Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
];

loss = Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"];

net = NetGraph[
{
net,
loss
},
{
NetPort["Input"] -> 1,
NetPort[1, "logvariance"] -> NetPort[2, "Input1"],
NetPort[1, "mean"] -> NetPort[2, "Input2"],
NetPort["Target"] -> NetPort[2, "Target"]
}
];

netT = NetTrain[
net,
<|
"Input" -> (Partition[#, 1] & /@ Standardize /@ enc@data[[;; , 1]]),
"Target" -> Partition[data[[;; , 2]], 1],
"Output" -> data[[;; , 2]]
|>,
MaxTrainingRounds -> 1
];

netT = NetExtract[netT, 1];

assoc = Options[p][[1]];
assoc["Model"]["Network"] = netT;
p1 = PredictorFunction[assoc];

pm1 = PredictorMeasurements[p1, data]
• This is great, but what about for my example in question (RNN on text)? I can't get it to work this way... thanks – user5601 Mar 19 '18 at 13:20
• @user5601 I was trying to make as simple example as possible. Answer to your question in addendum. – Alexey Golyshev Mar 20 '18 at 8:39
• Thanks for addressing my comment. I'm still unclear on what "UnitVectorLayer does not supported because of Standardize as the data processor" means? – user5601 Mar 20 '18 at 14:58
• Also what do the logvariance and mean NetPorts actually compute themselves, and why do we need to give the output to NetTrain? and must we use the loss NetGraph given by Predict, can't we define this arbitrarily? – user5601 Mar 20 '18 at 15:06
• 1. I found a way how UnitVectorLayer can be used. We can remove some data processors. But there is very strange error. 1drv.ms/u/s!AvVas0AkeAWi0GgEpOWYv9HPswtf – Alexey Golyshev Mar 20 '18 at 19:21