7
$\begingroup$

Consider a linear layer with 2*2 weights

linear = LinearLayer[2, "Input" -> 2];

net = NetInitialize[linear];

After the initialization, the four elements of the weight matrix W can be read

W = NetExtract[net, "Weights"]
(* {{0.551202, -1.10384}, {-0.659195, -1.12961}} *)

data = {{1, 1} -> {2, 3}, {2, 2} -> {4, 5.9}, {3, 3} -> {6, 9.1}, {4, 
     4} -> {8, 12.1}};

trainednet = NetTrain[net, data];

trainednet[{7, 7}]
(* {14., 21.25} *)

After training, the initial four weights W have been changed

NetExtract[trainednet, "Weights"]
(* {{1.82752, 0.17248}, {1.76021, 1.28979}} *)

Question: Is it possible to fix three of the weights (e.g., W[[1,1]]=1,W[[1,2]]=1,W[[2,1]]=1 ) so that only one weight (W[[2,2]]) can be changed during the training?

$\endgroup$
  • 1
    $\begingroup$ It will help if you provide additional input. May be a MWE, references, etc.... $\endgroup$ – Nasser Dec 5 '18 at 16:56
  • $\begingroup$ Thank Jason and Nasser. I have re-edited the question. $\endgroup$ – chrestun Dec 6 '18 at 8:21
  • $\begingroup$ I would restate the question as an ability to specify a hook for optimization step, in order to mask the gradient however I want. $\endgroup$ – swish Dec 6 '18 at 19:02
3
$\begingroup$
net = NetGraph[
   {
    ConstantArrayLayer[{3}, "Array" -> {1, 1, 1}],
    ConstantArrayLayer[{1}],
    CatenateLayer[],
    ReshapeLayer[{2, 2}],
    DotLayer[],
    2,
    SoftmaxLayer[]
    },
   {
    NetPort["Input"] -> 5, {1, 2} -> 3 -> 4 -> 5 -> 6 -> 7
    },
   "Input" -> 2,
   "Output" -> NetDecoder[{"Class", {0, 1}}]
   ] // NetInitialize

enter image description here

NetExtract[net, {1, "Array"}]

{1., 1., 1.}

NetExtract[net, {2, "Array"}]

{0.}

netT = NetTrain[
  net,
  RandomReal[{-1, 1}, {100, 2}] -> RandomInteger[{0, 1}, 100],
  LearningRateMultipliers -> {1 -> 0, _ -> 1},
  MaxTrainingRounds -> 10
  ]

enter image description here

NetExtract[netT, {1, "Array"}]

{1., 1., 1.}

NetExtract[netT, {2, "Array"}]

{-0.0881761}

$\endgroup$
  • $\begingroup$ It really is what I want. Thank you so much. $\endgroup$ – chrestun Dec 9 '18 at 14:58

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.