2
$\begingroup$

I've been playing around with the Higher-Order Network Construction functionality such as NetMapOperator, NetNestOperator, and NetFoldOperator in order to understand how they work. In particular I was trying to implement this differentiable model of morphogenesis using neural networks and cellular automata. I've cross-posted much of the workflow and results of this exploration in a Wolfram Community post. Here I just post the specific questions with minimally broken examples where applicable:

Q1) Can NetMapOperator take a 'level' argument like other Map-family functions? Shouldn't ElementwiseLayer also work to achieve the following?

NetMapOperator[NetMapOperator[NetChain[{2, Ramp}, "Input" -> 4]]]
Out[1]= NetMapOperator[ <> ]

ElementwiseLayer[NetChain[{2, Ramp}, "Input" -> 4]]
During evaluation of In[2]:= ElementwiseLayer::invscf: NetChain[<|Type->Chain,Nodes-><<1>>,Edges->{NetPath[Nodes,1,Inputs,Input]->NetPath[Inputs,Input],NetPath[Nodes,<<2>>,Input]-><<1>>,NetPath[Outputs,Output]->NetPath[Nodes,2,Outputs,Output]},Inputs-><|Input->TensorT[{4},AtomT]|>,Outputs-><|Output->TensorT[{2},RealT]|>|>,<<1>>] could not be symbolically evaluated as a unary scalar function.
Out[2]= $Failed

Q2) Is the Following expected behaviour or a bug (V12.2)? I.e. why does NetEvaluationMode cause the NetArrayLayer to get multiplied by a factor of 2?

droupout = 
  NetChain[{NetArrayLayer["Array" -> ConstantArray[1/2, {3, 3}], 
     LearningRateMultipliers -> 0], DropoutLayer[]}];
Tally[Flatten[droupout[]]]
Out[3]= {{0.5, 9}}

Tally[Flatten[droupout[NetEvaluationMode -> "Train"]]]
Out[4]= {{0., 4}, {1., 5}}

Q3) Can NetNestOperator take a non pre-defined value for the number of iterations? The following attempts with RandomArrayLayer and FunctionLayer were similarly fruitless:

nc = NetChain[{2, Ramp}];
FunctionLayer[Apply[Nest[nc, #1, #2] &]]
During evaluation of In[5]:= FunctionLayer::compilerr: Cannot interpret Nest[NetChain[<2>], #1, #2] & as a network.
Out[5]= $Failed

ra = RandomArrayLayer[DiscreteUniformDistribution[{64, 96}], 
   "Output" -> 1];
ncRA = NetChain[{ra, PartLayer[-1]}];
FunctionLayer[Nest[nc, #, ncRA] &]
During evaluation of In[6]:= FunctionLayer::compilerr: Cannot interpret Nest[NetChain[<2>], #1, NetChain[<2>]] & as a network.
Out[6]= $Failed

Q4) Suppose I wanted to train using Backpropagation in time, keeping a recurrent state, but without using the input port. I think I can hack it with the following. Is this correct? Is there a more convenient way to achieve this?

(*Reference NetNest implementation*)
nc = NetChain[{32, Ramp, 8, Ramp, 16, Ramp, 4}, "Input" -> 4];
nested = NetNestOperator[nc, 10];

(*Dummy Data*)
in = RandomReal[{0, 1}, {100, 4}];
out = {Sin[Max[#]], Cos[Min[#]], Cos[Min[#]]^2, Sin[Max[#]^2]} & /@ in;
nestedTrained = NetTrain[nested, <|"Input" -> in, "Output" -> out|>];

(*Fooling NetFoldOperator*)
dummyNC = 
  NetGraph[<|"times" -> ThreadingLayer[Times], 
    "nc" -> nc|>, {{NetPort["Input"], NetPort["State"]} -> 
     "times" -> "nc" -> NetPort["Output"]}, "Input" -> "Scalar"];

dummyFolded = 
  NetGraph[<|"fold" -> NetFoldOperator[dummyNC], 
    "loss" -> MeanSquaredLossLayer[], 
    "state-seq" -> SequenceLastLayer[]|>, {NetPort["InitialState"] -> 
     NetPort["fold", "State"], 
    NetPort["fold", "Output"] -> 
     "state-seq" -> NetPort["Final State"], {"state-seq", 
      NetPort["Target"]} -> "loss"}];

dummyFoldedTrained = 
 NetTrain[dummyFolded, <|
   "Input" -> 
    Table[ConstantArray[1, {RandomInteger[{8, 12}], 1}], 100], 
   "InitialState" -> in, "Target" -> out|>]

Note the sequences are no - longer length 10 but each is a random integer b/w 8 - 12

$\endgroup$
2
  • 1
    $\begingroup$ Q2: Dropout sets the input elements to zero with probability p during training, multiplying the remainder by 1/(1-p). Default probability is 0.5. 1/(1-0.5)=2 $\endgroup$ Commented Jan 2, 2021 at 15:04
  • $\begingroup$ Oh thanks, must've missed this.. I see it now under Method in Details and Options! I wonder if the additional clause "multiplying the remainder by 1/(1-p)" should be stated in Information[DropoutLayer, "Usage"] as-well $\endgroup$ Commented Jan 2, 2021 at 15:08

0