14
$\begingroup$

PredictorMeasurements doesn't work with NetGraph, here's an example:

makeRule[a_, b_] := 
  IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}];
enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];
net = NetInitialize@
   NetChain[{UnitVectorLayer[], LongShortTermMemoryLayer[40], 
     LongShortTermMemoryLayer[20], SequenceLastLayer[], 
     LinearLayer[]}, "Input" -> enc, "Output" -> "Real"];
PredictorMeasurements[net, data, "Accuracy"]

enter image description here

Is there any way to make this work? Perhaps converting the net into a predictor?

$\endgroup$
0

1 Answer 1

7
+200
$\begingroup$

Let's look under the hood of Predict.

p = Predict[{{1, 2} -> 3, {2, 3} -> 4}, 
   Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"}];

Options[p][[1]]["Model"]["Network"]

The network has 2 outputs: mean and log-variance.

enter image description here

Options[p][[1]]["Model"]["Options"]["Network"]["Value"]

enter image description here

Loss function is very interesting:

Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"]

enter image description here

And now let's replace trained network in Predict with our custom net.

net = NetGraph[
   {
    LongShortTermMemoryLayer[40],
    NetMapOperator[LinearLayer[10]],
    LongShortTermMemoryLayer[20],
    SequenceLastLayer[],
    LinearLayer[100],
    Ramp,
    LinearLayer[2],
    PartLayer[1 ;; 1],
    PartLayer[2 ;; 2]
    },
   {1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> {8, 9}, 
    8 -> NetPort["logvariance"], 9 -> NetPort["mean"]},
   "Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
   ] // NetInitialize

enter image description here

GeneralUtilities`PrintDefinitions@PredictorFunction

We can see that PredictorFunction expects Association as the input.

assoc = Options[p][[1]];
assoc["Model"]["Network"] = net;
p1 = PredictorFunction[assoc]

enter image description here

We can make predictions:

p1[{{1, 2}, {2, 3}}]

{3.49756, 3.50435}

And we can do PredictorMeasurements:

pm1 = PredictorMeasurements[p1, {{1, 2} -> 3, {2, 3} -> 4}]

enter image description here

pm1["MeanSquare"]

0.246618

Addendum

makeRule[a_, b_] := IntegerString[a] <> "+" <> IntegerString[b] -> a + b;
data = Table[makeRule[i, j], {i, 0, 99}, {j, 0, 99}] // Flatten;

enc = NetEncoder[{"Characters", {DigitCharacter, "+"}}];

fe = FeatureExtraction[data[[;; , 1]], enc];

p = Predict[
   data[[-2 ;;, 1]] -> data[[-2 ;;, 2]],
   Method -> {"NeuralNetwork", "NetworkType" -> "Recurrent"},
   FeatureExtractor -> fe
   ];

net = NetGraph[
  {
   (* UnitVectorLayer does not supported because of Standardize as the data processor *)
   LongShortTermMemoryLayer[40],
   LongShortTermMemoryLayer[20],
   SequenceLastLayer[],
   LinearLayer[2],
   PartLayer[1 ;; 1],
   PartLayer[2 ;; 2]
   },
  {1 -> 2 -> 3 -> 4 -> {5, 6}, 5 -> NetPort["logvariance"], 6 -> NetPort["mean"]}, 
  "Input" -> {"Varying", 1}, "logvariance" -> 1, "mean" -> 1
  ];

loss = Options[p][[1]]["Model"]["Options"]["LossFunction"]["Value"];

net = NetGraph[
   {
    net,
    loss
    },
   {
    NetPort["Input"] -> 1,
    NetPort[1, "logvariance"] -> NetPort[2, "Input1"],
    NetPort[1, "mean"] -> NetPort[2, "Input2"],
    NetPort["Target"] -> NetPort[2, "Target"]
    }
   ];

netT = NetTrain[
   net,
   <|
    "Input" -> (Partition[#, 1] & /@ Standardize /@ enc@data[[;; , 1]]),
    "Target" -> Partition[data[[;; , 2]], 1],
    "Output" -> data[[;; , 2]]
    |>,
   MaxTrainingRounds -> 1
   ];

netT = NetExtract[netT, 1];

assoc = Options[p][[1]];
assoc["Model"]["Network"] = netT;
p1 = PredictorFunction[assoc];

pm1 = PredictorMeasurements[p1, data]
$\endgroup$
8
  • $\begingroup$ This is great, but what about for my example in question (RNN on text)? I can't get it to work this way... thanks $\endgroup$
    – user5601
    Mar 19, 2018 at 13:20
  • 1
    $\begingroup$ @user5601 I was trying to make as simple example as possible. Answer to your question in addendum. $\endgroup$ Mar 20, 2018 at 8:39
  • $\begingroup$ Thanks for addressing my comment. I'm still unclear on what "UnitVectorLayer does not supported because of Standardize as the data processor" means? $\endgroup$
    – user5601
    Mar 20, 2018 at 14:58
  • $\begingroup$ Also what do the logvariance and mean NetPorts actually compute themselves, and why do we need to give the output to NetTrain? and must we use the loss NetGraph given by Predict, can't we define this arbitrarily? $\endgroup$
    – user5601
    Mar 20, 2018 at 15:06
  • $\begingroup$ 1. I found a way how UnitVectorLayer can be used. We can remove some data processors. But there is very strange error. 1drv.ms/u/s!AvVas0AkeAWi0GgEpOWYv9HPswtf $\endgroup$ Mar 20, 2018 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.