1
$\begingroup$

Trying to design a small neural network for modeling a LTI system, I'd like to configure a BasicRecurrentLayer with these particular settings:

  • No bias vector should be used (I've tried the option "Biases"-> None, but it does not work for a BasicRecurrentLayer)
  • The output should be directly the weighted sum of its inputs (I'd like to use Identity instead of Tanh as activation function, which is the default value for this kind of layer)

I could apply a ElementwiseLayer[ArcTanh] at the output of the layer, to somehow undo the effect of Tanh, but still the recurrent connection going backwards to the input port would not be the desired one.

As far as I can see, I can't configure BasicRecurrentLayer to my own needs, so must I then create a new user-defined layer, and make the necessary recurrent connections manually into a NetGraph? How could I do it? I'm a newbie in the neural networks field, and any help is much appreciated.

Thanks in advance, kind regards,

$\endgroup$
1
  • $\begingroup$ Could you please include the code you have developed so far in your attempts? That would give a valuable starting point for people to play around with and help you with your problem. $\endgroup$
    – MarcoB
    Mar 4, 2022 at 13:18

1 Answer 1

0
$\begingroup$
SeedRandom[0];
batchSize = 16;
X = RandomReal[{-1, 1}, {batchSize, 10, 8}];
Y = RandomInteger[{0, 1}, {batchSize, 1}];
net = NetChain[
   {
    BasicRecurrentLayer[4],
    SequenceLastLayer[],
    LinearLayer[1],
    ElementwiseLayer[LogisticSigmoid]
    },
   "Input" -> {"Varying", 8}
   ] // NetInitialize

enter image description here

NetExtract[net, {1, "Biases"}] // Normal

{0., 0., 0., 0.}

netT = NetTrain[net, X -> Y, MaxTrainingRounds -> 10];

NetExtract[netT, {1, "Biases"}] // Normal

{0.00993016, 0.00993503, -0.00995534, -0.00985054}

LearningRateMultipliers is useful here.

netT2 = NetTrain[net, X -> Y, MaxTrainingRounds -> 10, 
   LearningRateMultipliers -> {{1, "Biases"} -> 0}];

NetExtract[netT2, {1, "Biases"}] // Normal

{0., 0., 0., 0.}

You can also look at NetFoldOperator to create customized RNN.

$\endgroup$
2
  • $\begingroup$ OK for the method to get rid of the biases, but why did you add a LogisticSigmoid at the end of the chain? I just need a linear function at the output. $\endgroup$
    – Danel
    Mar 16, 2022 at 8:26
  • $\begingroup$ Hello @Danel. You can delete LogisticSigmoid. In this example, Y is in the range [0, 1]. $\endgroup$ Mar 16, 2022 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.