Loss function with regularization

Question

The ML framework has a limited set of loss functions available. How can I extend this by creating my own custom ones?

For example, for L2-regularized training, Mathematica recommends

NetTrain[
  net, data, All, 
  Method -> {"SGD", "L2Regularization" -> 0.01}
]

Question: How is L2Regularization implemented here? Is there a way to probe how Mathematica implements this and the functional form of the loss function?

Instead, I would like to have a loss function that has L2-regularization: $$loss = MSE + \lambda(||parameter||_2) $$

This would be a starting point to play around with different kinds of regularization.

As another example (Example 3.2 : https://reference.wolfram.com/language/ref/LossFunction.html), they created a MSE loss function; I want to add regularization (l1 and l2) to it.

Edit: To make it precise, how can I add regularization in the loss net below:

lossNet = 
 NetGraph[<|"net" -> net, "loss" -> ThreadingLayer[(#1 - #2)^2 &]|>, {{"net", NetPort["Target"]} ->  "loss" -> NetPort["Loss"]}]

Answer 1

Since version 12.2, you can use NetArray and NetArrayLayer for this.

For example:

\[Lambda] = 1*^-4;
trynetgraph = NetGraph[<|
   "network" -> 
    LinearLayer[10, "Weights" -> NetArray["Name" -> "netWeights"]],
   "netWeightsRegularizaton" -> 
    NetChain[{NetArrayLayer[
       "Array" -> NetArray["Name" -> "netWeights"], 
       "Output" -> {10, 2*2}], ElementwiseLayer[\[Lambda]*#^2 &], 
      SummationLayer[]}],
   "MSE" -> MeanSquaredLossLayer[],
   "TotalLayer" -> TotalLayer[]
   |>, {
   NetPort["Target"] -> NetPort["MSE", "Target"],
   "network" -> NetPort["MSE", "Input"],
   {"MSE", "netWeightsRegularizaton"} -> 
    "TotalLayer" -> NetPort["Loss"]
   }]

The dimensions for the NetArrayLayer had to be put in manually, since as for the current version, automatic dimension inference doesn't seem to work well for this case.

Training this network with random data:

ramdat = Table[<|"Input" -> RandomReal[{}, {2, 2}], 
        "Target" -> RandomReal[{}, {10}]|>, {n, 100}];
NetTrain[trynetgraph, ramdat, LossFunction -> "Loss"]