11
$\begingroup$

Basically one creates a NetChain of several layers, trains it and by evaluating model[testdata], the prediction can be obtained. However, I am unable to obtain the intermediate values of the prediction, i.e., the output after each layer with basic function usages.

I have tried to build a NetGraph, creating a path from the node I want to inspect to another output, but it didn't work since all the outputs must be a part of training process.

I also tried NetExtract, and succeed only with simple structured networks. Here is an small example to illustrate my question.

(*build an RNN with simple layers*)
test = NetInitialize[
  NetChain[{LongShortTermMemoryLayer[5], SequenceLastLayer[], 
    LinearLayer[2], SoftmaxLayer[]}, "Input" -> {"Varying", 3}], 
  Method -> "Random"];
(*make a prediction*)
test[{{1, 2, 3}, {0, 1, 3}}]
(*{0.299767, 0.700233}*)
(*use net extract and manually evaluate the model*)
ComposeList[NetExtract[test, #] & /@ Range[4], {{1, 2, 3}, {0, 1, 3}}]
(*{{{1, 2, 3}, {0, 1, 
   3}}, {{0.279698, -0.0118263, -0.000721162, -0.142595, \
-0.00157304}, {0.244337, -0.149762, -0.000606668, -0.138503, \
-0.0288229}}, {0.244337, -0.149762, -0.000606668, -0.138503, \
-0.0288229}, {-0.638368, 0.210041}, {0.299767, 0.700233}}*)

In my actual problem, I need to capture the probabilities of a classifier before it is encoded to classes. ClassifierMeasurements works for single-output classifiers, but mine outputs a sequence of classes and can't be processed by that function.

Is there a convenient/native way of doing this?

$\endgroup$
7
$\begingroup$

You can also tap off the intermediate layers by attaching output ports. For example

net = NetInitialize@
  NetGraph[{LongShortTermMemoryLayer[5], SequenceLastLayer[], 
    LinearLayer[2], 
    SoftmaxLayer[]}, {NetPort["Input"] -> 
     1 -> 2 -> 3 -> 4 -> NetPort["Output"], 1 -> NetPort["layer1"], 
    2 -> NetPort["layer2"], 3 -> NetPort["layer3"]}, 
   "Input" -> {"Varying", 3}]

enter image description here

Then

net[{{1, 2, 3}, {0, 1, 3}}]
(* <|"Output" -> {0.716093, 0.283907}, 
 "layer1" -> {{0.0242801, 0.00128676, -0.220564, 0.010784, 
    0.369972}, {-0.0774157, 0.0137297, -0.456915, 0.0116218, 
    0.665373}}, 
 "layer2" -> {-0.0774157, 0.0137297, -0.456915, 0.0116218, 0.665373}, 
 "layer3" -> {0.748411, -0.176754}|> *)

You don't have to use all the output ports in the training process. For example, this uses only the original input and output ports:

NetTrain[net, {<|"Input" -> {{1, 2, 3}, {0, 1, 3}}, 
   "Output" -> {0.5281522274017334`, 0.471847802400589`}|>},
   "Output" -> MeanSquaredLossLayer[]]

This works for NetGraph as well.


Here is a complete example using MNIST where the 10th layer is tapped off:

lenet = NetGraph[{
   ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
   ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
   FlattenLayer[], 500, Ramp, 10, 
   SoftmaxLayer[]}, {NetPort["Input"] -> 
    1 -> 2 -> 
      3 -> 4 -> 
        5 -> 6 -> 7 -> 8 -> 9 -> 10 -> 11 -> NetPort["Output"], 
   10 -> NetPort["layer10"]},
  "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}], 
  "Output" -> NetDecoder[{"Class", Range[0, 9]}]
  ]

trainingData2 = <|"Input" -> #[[1]], "Output" -> #[[2]]|> & /@ 
   RandomSample[trainingData, 1000];

trained = NetTrain[lenet, trainingData2, "Output" -> CrossEntropyLossLayer["Index"]]

trained[trainingData2[[1, 1]]]
(* <|"Output" -> 5, 
 "layer10" -> {-2.34726, -9.10603, -0.501655, -0.602975, 0.368083, 
   6.48835, -5.21255, -1.2072, 1.68743, 5.18232}|> *)
$\endgroup$
  • $\begingroup$ I haven't tried this, but @happyfish said in the question: I have tried to build a NetGraph, creating a path from the node I want to inspect to another output, but it didn't work since all the outputs must be a part of training process. Isn't that what you're doing? $\endgroup$ – Niki Estner Apr 21 '17 at 16:11
  • $\begingroup$ Sorry, I missed that part in the first place. I've added the corrections. $\endgroup$ – xslittlegrass Apr 21 '17 at 16:24
  • 2
    $\begingroup$ "Output" -> MeanSquaredLossLayer[] that's the part I missed. The documentation says "port"->losslayer compute loss from one output of a multi-output net, which is not clear about what's used to train. Your elaboration revealed its actual usage and this is exactly what I needed. Thanks a lot! $\endgroup$ – happy fish Apr 22 '17 at 13:07
  • $\begingroup$ I tried xslittlegrass's solution, But it didn’t work and I got error massages: OptionValue::nodef "Unknown option \!(\"Output\") for NetTrain NetTrain::missinslot "Specification for slot \!(\"layer5\") is missing. Any ideas ? I'm using Mathematica 11.3 Tnank you! $\endgroup$ – GM_jnj Jan 17 at 12:31
  • $\begingroup$ Please open a new question with your details. $\endgroup$ – Carl Lange Jan 17 at 13:30
6
$\begingroup$

Not really "convenient", but you can get the layers of a NetChain using Normal:

net = NetInitialize@NetChain[{3, Ramp, 4, Ramp, 1}, "Input" -> "Real"];
layers = Normal[net]

and then construct a NetGraph with the same layers, and an output port for each layer:

g = NetGraph[layers,
  Join[
   (* connections between adjacent layers *)
   # -> # + 1 & /@ Range[Length[layers] - 1],
   (* output ports *)
   # -> NetPort[StringTemplate["Layer_``"][#]] & /@ 
    Range[Length[layers]]
   ]]

Result:

net[1]

{0.168289}

g[1]

<|"Layer_1" -> {-0.792354, 0.213135, 0.565724}, "Layer_2" -> {0., 0.213135, 0.565724}, "Layer_3" -> {-0.422488, 0.126484, 0.000964101, 0.0990122}, "Layer_4" -> {0., 0.126484, 0.000964101, 0.0990122}, "Layer_5" -> {0.168289}|>

$\endgroup$
  • $\begingroup$ Thanks! I have got your point. Before checking your answer, I will wait some more hours to see if someone knows (or there exists) some "builtin" functions to do this. $\endgroup$ – happy fish Apr 21 '17 at 15:38
  • $\begingroup$ @happyfish: Absolutely. I also hope there's a simpler solution, one that works for NetGraph networks, too. $\endgroup$ – Niki Estner Apr 21 '17 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.