5
$\begingroup$

The ML framework has a limited set of loss functions available. How can I extend this by creating my own custom ones?

For example, for L2-regularized training, Mathematica recommends

NetTrain[
  net, data, All, 
  Method -> {"SGD", "L2Regularization" -> 0.01}
]

Question: How is L2Regularization implemented here? Is there a way to probe how Mathematica implements this and the functional form of the loss function?

Instead, I would like to have a loss function that has L2-regularization: $$loss = MSE + \lambda(||parameter||_2) $$

This would be a starting point to play around with different kinds of regularization.

As another example (Example 3.2 : https://reference.wolfram.com/language/ref/LossFunction.html), they created a MSE loss function; I want to add regularization (l1 and l2) to it.

Edit: To make it precise, how can I add regularization in the loss net below:

lossNet = 
 NetGraph[<|"net" -> net, "loss" -> ThreadingLayer[(#1 - #2)^2 &]|>, {{"net", NetPort["Target"]} ->  "loss" -> NetPort["Loss"]}]
$\endgroup$
4
  • 2
    $\begingroup$ FindFit in version 12 offers FitRegularization as an option. $\endgroup$ – JimB Feb 25 at 4:39
  • $\begingroup$ Thanks @JimB for the comments. How does FitRegularization help me construct loss function for neural net? $\endgroup$ – psimeson Feb 25 at 5:55
  • $\begingroup$ It likely doesn't help with neural net. I mention it only because looks like Mathematica is starting to add that option to some functions. (It still isn't available for LinearModelFit or NonlinearModelFit.) $\endgroup$ – JimB Feb 25 at 6:24
  • $\begingroup$ This is what Mathematica technical support told me: "It is currently not possible to add the regularization term in the loss function itself without specifying in Method. Only L2-Regularization is available as a suboption of Method." $\endgroup$ – psimeson Mar 10 at 17:29
4
$\begingroup$

Since version 12.2, you can use NetArray and NetArrayLayer for this.

For example:

\[Lambda] = 1*^-4;
trynetgraph = NetGraph[<|
   "network" -> 
    LinearLayer[10, "Weights" -> NetArray["Name" -> "netWeights"]],
   "netWeightsRegularizaton" -> 
    NetChain[{NetArrayLayer[
       "Array" -> NetArray["Name" -> "netWeights"], 
       "Output" -> {10, 2*2}], ElementwiseLayer[\[Lambda]*#^2 &], 
      SummationLayer[]}],
   "MSE" -> MeanSquaredLossLayer[],
   "TotalLayer" -> TotalLayer[]
   |>, {
   NetPort["Target"] -> NetPort["MSE", "Target"],
   "network" -> NetPort["MSE", "Input"],
   {"MSE", "netWeightsRegularizaton"} -> 
    "TotalLayer" -> NetPort["Loss"]
   }]

NetGraph Diagram

The dimensions for the NetArrayLayer had to be put in manually, since as for the current version, automatic dimension inference doesn't seem to work well for this case.

Training this network with random data:

ramdat = Table[<|"Input" -> RandomReal[{}, {2, 2}], 
        "Target" -> RandomReal[{}, {10}]|>, {n, 100}];
NetTrain[trynetgraph, ramdat, LossFunction -> "Loss"]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.