12
$\begingroup$

PredictorMeasurements doesn't work with NetGraph, here's an example:

makeRule[a_, b_] := 
  IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}];
enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];
net = NetInitialize@
   NetChain[{UnitVectorLayer[], LongShortTermMemoryLayer[40], 
     LongShortTermMemoryLayer[20], SequenceLastLayer[], 
     LinearLayer[]}, "Input" -> enc, "Output" -> "Real"];
PredictorMeasurements[net, data, "Accuracy"]

enter image description here

Is there any way to make this work? Perhaps converting the net into a predictor?

$\endgroup$
5
+200
$\begingroup$

Let's look under the hood of Predict.

p = Predict[{{1, 2} -> 3, {2, 3} -> 4}, 
   Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"}];

Options[p][[1]]["Model"]["Network"]

The network has 2 outputs: mean and log-variance.

enter image description here

Options[p][[1]]["Model"]["Options"]["Network"]["Value"]

enter image description here

Loss function is very interesting:

Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"]

enter image description here

And now let's replace trained network in Predict with our custom net.

net = NetGraph[
   {
    LongShortTermMemoryLayer[40],
    NetMapOperator[LinearLayer[10]],
    LongShortTermMemoryLayer[20],
    SequenceLastLayer[],
    LinearLayer[100],
    Ramp,
    LinearLayer[2],
    PartLayer[1 ;; 1],
    PartLayer[2 ;; 2]
    },
   {1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> {8, 9}, 
    8 -> NetPort["logvariance"], 9 -> NetPort["mean"]},
   "Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
   ] // NetInitialize

enter image description here

GeneralUtilities`PrintDefinitions@PredictorFunction

We can see that PredictorFunction expects Association as the input.

assoc = Options[p][[1]];
assoc["Model"]["Network"] = net;
p1 = PredictorFunction[assoc]

enter image description here

We can make predictions:

p1[{{1, 2}, {2, 3}}]

{3.49756, 3.50435}

And we can do PredictorMeasurements:

pm1 = PredictorMeasurements[p1, {{1, 2} -> 3, {2, 3} -> 4}]

enter image description here

pm1["MeanSquare"]

0.246618

Addendum

makeRule[a_, b_] := IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}] // Flatten;

enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];

fe = FeatureExtraction[data[[;; , 1]], enc];

p = Predict[
   data[[-2 ;;, 1]] -> data[[-2 ;;, 2]],
   Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"},
   FeatureExtractor -> fe
   ];

net = NetGraph[
  {
   (* UnitVectorLayer does not supported because of Standardize as the data processor *)
   LongShortTermMemoryLayer[40],
   LongShortTermMemoryLayer[20],
   SequenceLastLayer[],
   LinearLayer[2],
   PartLayer[1 ;; 1],
   PartLayer[2 ;; 2]
   },
  {1 -> 2 -> 3 -> 4 -> {5, 6}, 5 -> NetPort["logvariance"], 6 -> NetPort["mean"]}, 
  "Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
  ];

loss = Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"];

net = NetGraph[
   {
    net,
    loss
    },
   {
    NetPort["Input"] -> 1,
    NetPort[1, "logvariance"] -> NetPort[2, "Input1"],
    NetPort[1, "mean"] -> NetPort[2, "Input2"],
    NetPort["Target"] -> NetPort[2, "Target"]
    }
   ];

netT = NetTrain[
   net,
   <|
    "Input" -> (Partition[#, 1] & /@ Standardize /@ enc@data[[;; , 1]]),
    "Target" -> Partition[data[[;; , 2]], 1],
    "Output" -> data[[;; , 2]]
    |>,
   MaxTrainingRounds -> 1
   ];

netT = NetExtract[netT, 1];

assoc = Options[p][[1]];
assoc["Model"]["Network"] = netT;
p1 = PredictorFunction[assoc];

pm1 = PredictorMeasurements[p1, data]
$\endgroup$
  • $\begingroup$ This is great, but what about for my example in question (RNN on text)? I can't get it to work this way... thanks $\endgroup$ – user5601 Mar 19 '18 at 13:20
  • 1
    $\begingroup$ @user5601 I was trying to make as simple example as possible. Answer to your question in addendum. $\endgroup$ – Alexey Golyshev Mar 20 '18 at 8:39
  • $\begingroup$ Thanks for addressing my comment. I'm still unclear on what "UnitVectorLayer does not supported because of Standardize as the data processor" means? $\endgroup$ – user5601 Mar 20 '18 at 14:58
  • $\begingroup$ Also what do the logvariance and mean NetPorts actually compute themselves, and why do we need to give the output to NetTrain? and must we use the loss NetGraph given by Predict, can't we define this arbitrarily? $\endgroup$ – user5601 Mar 20 '18 at 15:06
  • $\begingroup$ 1. I found a way how UnitVectorLayer can be used. We can remove some data processors. But there is very strange error. 1drv.ms/u/s!AvVas0AkeAWi0GgEpOWYv9HPswtf $\endgroup$ – Alexey Golyshev Mar 20 '18 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.