I've been playing around with the Higher-Order Network Construction functionality such as NetMapOperator
, NetNestOperator
, and NetFoldOperator
in order to understand how they work. In particular I was trying to implement this differentiable model of morphogenesis using neural networks and cellular automata. I've cross-posted much of the workflow and results of this exploration in a Wolfram Community post.
Here I just post the specific questions with minimally broken examples where applicable:
Q1) Can NetMapOperator
take a 'level' argument like other Map
-family functions? Shouldn't ElementwiseLayer
also work to achieve the following?
NetMapOperator[NetMapOperator[NetChain[{2, Ramp}, "Input" -> 4]]]
Out[1]= NetMapOperator[ <> ]
ElementwiseLayer[NetChain[{2, Ramp}, "Input" -> 4]]
During evaluation of In[2]:= ElementwiseLayer::invscf: NetChain[<|Type->Chain,Nodes-><<1>>,Edges->{NetPath[Nodes,1,Inputs,Input]->NetPath[Inputs,Input],NetPath[Nodes,<<2>>,Input]-><<1>>,NetPath[Outputs,Output]->NetPath[Nodes,2,Outputs,Output]},Inputs-><|Input->TensorT[{4},AtomT]|>,Outputs-><|Output->TensorT[{2},RealT]|>|>,<<1>>] could not be symbolically evaluated as a unary scalar function.
Out[2]= $Failed
Q2) Is the Following expected behaviour or a bug (V12.2)? I.e. why does NetEvaluationMode
cause the NetArrayLayer
to get multiplied by a factor of 2?
droupout =
NetChain[{NetArrayLayer["Array" -> ConstantArray[1/2, {3, 3}],
LearningRateMultipliers -> 0], DropoutLayer[]}];
Tally[Flatten[droupout[]]]
Out[3]= {{0.5, 9}}
Tally[Flatten[droupout[NetEvaluationMode -> "Train"]]]
Out[4]= {{0., 4}, {1., 5}}
Q3) Can NetNestOperator
take a non pre-defined value for the number of iterations? The following attempts with RandomArrayLayer
and FunctionLayer
were similarly fruitless:
nc = NetChain[{2, Ramp}];
FunctionLayer[Apply[Nest[nc, #1, #2] &]]
During evaluation of In[5]:= FunctionLayer::compilerr: Cannot interpret Nest[NetChain[<2>], #1, #2] & as a network.
Out[5]= $Failed
ra = RandomArrayLayer[DiscreteUniformDistribution[{64, 96}],
"Output" -> 1];
ncRA = NetChain[{ra, PartLayer[-1]}];
FunctionLayer[Nest[nc, #, ncRA] &]
During evaluation of In[6]:= FunctionLayer::compilerr: Cannot interpret Nest[NetChain[<2>], #1, NetChain[<2>]] & as a network.
Out[6]= $Failed
Q4) Suppose I wanted to train using Backpropagation in time, keeping a recurrent state, but without using the input port. I think I can hack it with the following. Is this correct? Is there a more convenient way to achieve this?
(*Reference NetNest implementation*)
nc = NetChain[{32, Ramp, 8, Ramp, 16, Ramp, 4}, "Input" -> 4];
nested = NetNestOperator[nc, 10];
(*Dummy Data*)
in = RandomReal[{0, 1}, {100, 4}];
out = {Sin[Max[#]], Cos[Min[#]], Cos[Min[#]]^2, Sin[Max[#]^2]} & /@ in;
nestedTrained = NetTrain[nested, <|"Input" -> in, "Output" -> out|>];
(*Fooling NetFoldOperator*)
dummyNC =
NetGraph[<|"times" -> ThreadingLayer[Times],
"nc" -> nc|>, {{NetPort["Input"], NetPort["State"]} ->
"times" -> "nc" -> NetPort["Output"]}, "Input" -> "Scalar"];
dummyFolded =
NetGraph[<|"fold" -> NetFoldOperator[dummyNC],
"loss" -> MeanSquaredLossLayer[],
"state-seq" -> SequenceLastLayer[]|>, {NetPort["InitialState"] ->
NetPort["fold", "State"],
NetPort["fold", "Output"] ->
"state-seq" -> NetPort["Final State"], {"state-seq",
NetPort["Target"]} -> "loss"}];
dummyFoldedTrained =
NetTrain[dummyFolded, <|
"Input" ->
Table[ConstantArray[1, {RandomInteger[{8, 12}], 1}], 100],
"InitialState" -> in, "Target" -> out|>]
Note the sequences are no - longer length 10 but each is a random integer b/w 8 - 12
Method
in Details and Options! I wonder if the additional clause "multiplying the remainder by 1/(1-p)" should be stated inInformation[DropoutLayer, "Usage"]
as-well $\endgroup$