Is there a way to create our own custom layers for Neural Networks in Mathematica ? I suspect it should be possible, any pointers or directions on how to go about it?

I am interested in functionality like the "Lambda Layer" in Keras API.

  • 11
    $\begingroup$ I would love to see some docs on NeuralNetworks`DefineLayer $\endgroup$ – swish Feb 20 '17 at 20:18
  • $\begingroup$ Thats interesting ... $\endgroup$ – Andy Stow Away Feb 20 '17 at 20:23
  • 1
    $\begingroup$ @swish Can you give a example that success to use it? $\endgroup$ – yode Feb 21 '17 at 2:17
  • $\begingroup$ @yode Don't have one, that is the problem. $\endgroup$ – swish Feb 21 '17 at 10:07
  • $\begingroup$ I am looking for a functionality that makes a layer can also let its weights be part of the output, so that one can use them to define a more sophisticated loss function, e.g., imposing constraints on the weights including but not limited to sum of absolute squares. $\endgroup$ – Αλέξανδρος Ζεγγ Mar 28 '18 at 6:18

Supporting custom layers is on our to-do list, and should be ready for either 11.2 or 11.3.

For interest: what applications do you want custom layers for? And how performant do you need your custom layer to be? (for example, do you want to be able to write your own CUDA implementation? Or are you happy with writing your own CPU layer?)

  • 1
    $\begingroup$ Not a custom layer per se, but the ability to costomize weight sharing would be really useful. $\endgroup$ – Sascha Mar 26 '17 at 20:12
  • 1
    $\begingroup$ @AlexeyGolyshev: Mathematica is designed to be a functional language where things don't usually have state. Our network framework is built on the same principle. This means that we expose the RNN states explicitly as arguments (see the docs for LSTM etc). Its definitely annoying to use this for generation from RNN's. Expect major useability improvements for generation in 11.2. Also phased-lstm: will take a look. Can you implement it in 11.1 using NetFoldOperator? $\endgroup$ – Sebastian Mar 28 '17 at 10:15
  • 3
    $\begingroup$ Sebastian, thanks for your response. I am using Mathematica 11.3 and am wondering if this functionality has been actually implemented. One reason to use it is to be able to take advantage of the rest of the system while investigating your own ideas in the NN area. By the way, thanks for the great article on O'Reilly! $\endgroup$ – LevK Mar 28 '18 at 19:34
  • 4
    $\begingroup$ @levK: I implemented a general custom-layer mechanism some months ago, but it turned out to cause many problems due to the asynchronous nature of the callbacks. For 12, a really good solution should exist though: custom layers via using the upcoming compiler infrastructure, with the end-game being direct compilation to CUDA! So stay tuned :-) $\endgroup$ – Sebastian Apr 4 '18 at 8:48
  • 2
    $\begingroup$ @Sebastian did this ever happen? $\endgroup$ – user5601 Sep 12 '19 at 5:22

Here's my guess for how to call DefineLayer based on the DownValues and some minimal spelunking.

First we define an Association defining the layer à la NeuralNetworks`$LayerData (seems that you need to drop {"Symbol", "Type"} based on how I was calling it):

 KeyDrop[ NeuralNetworks`$LayerData["Linear"], {"Symbol", "Type"}]

Then just give it a name as a String:

NeuralNetworks`DefineLayer["Fake", fakeAssoc];

Now NeuralNetworks`FakeLayer is defined:


fake layer

If we want a non "NeuralNetworks`" level symbol instead, we'd need to make sure the NameQ["System`"<>name] is `True:

  KeyDrop[ NeuralNetworks`$LayerData["Linear"], {"Symbol", "Type"}]


fake 2

I think NeuralNetworks`$LayerData["Linear"] should be clear enough as a template for doing this. The parameter names seem pretty straight-forward:

NeuralNetworks`$LayerData["Linear"] // Keys

{"Inputs", "Outputs", "States", "Arrays", "Parameters", \
"ParameterCoercions", "ParameterDefaults", "InferenceRules", \
"RuntimeInferenceRules", "ShapeFunction", "RankFunction", \
"ExtraShapeFunctionTensors", "PostInferenceFunction", \
"PostConstructionFunction", "AuxArrays", "Writer", "Upgraders", \
"MXNet", "SubNets", "IsLoss", "IsOperator", "IsMultiport", \
"InheritsFrom", "Suffix", "SummaryFunction", "Tests", \
"AllowDynamicDimensions", "ReshapeParams", "ArgumentRewriter", \
"CUDNNArray", "Constraints", "StateExpanding", "MaxArgCount", \
"MinArgCount", "PosArgCount", "PosArgs", "SourceFile", "Type", \

There are some documents written in markdown, under the path SystemFiles\Components\NeuralNetworks. Those under Layers are particularly of interest to question in OP.

       , "SystemFiles", "Components", "NeuralNetworks"
       }] // FileNames["*.md", #, ∞] &

                         .  .  .     \Layers\LayerDefinitions.md
                         .  .  .     \Layers\ShapePolymorphism.md
                         .  .  .     \Layers\Writers.md
                         .  .  .     \MXNet\Evaluation.md
                         .  .  .     \NetTrain.m\InternalOptions.md
                         .  .  .     \NetTrain.m\Overview.md
                         .  .  .     \Organization.md

Here is a practical example using NeuralNetworks`DefineLayer: ModelZoo/PGGAN-128 trained on Anime

PixelNorm.m defines a newPixelNormalizationLayer:

Input: ChannelT[$$Channels, TensorT[$$InputDimensions]]
Output: ChannelT[$$Channels, TensorT[$$InputDimensions]]

    $Epsilon: Defaulting[ScalarT, 10^-8]
    $$Channels: SizeT
    $$InputDimensions: SizeListT[SizeT]

Writer: Function[
    input = GetInput["Input", "Batchwise"];
    path = SowNode["mean", SowSquare@input, "axis" -> 1, "keepdims" -> True];
    path = SowRSqrt@SowNode["_PlusScalar", path, "scalar" -> #Epsilon];
    output = SowNode["broadcast_mul", {input, path}];
    SetOutput["Output", output]

Suffix: "alizationLayer"

enter image description here

The executable notebook can be downloaded from PGGAN-128.trained.on.Anime.zip

Every time you load the network, you must first load the custom layers through NeuralNetworks`DefineLayer.

Such a network cannot be submitted to NeuralNetRepository, and currently there is no solution similar to FunctionRepository.

You can see that the custom layer uses a domain-specific language

All options can be viewed by NeuralNetworks`Private`DefineLayer`LayerDefinitionT[[1, All, 1]]

There also contains type and interface information.

These options have their own roles, but there are only two necessary parts of a layer: Input/Output and Writer.

If you don't want to check the input and output, you can directly set it to AnyTensorT, which will not do any shape check.

Writer can specify how to convert to sow node.

Sow is a number of predefined operators.You can view all definitions via ? NeuralNetworks`Private`Sow*.

If it is not defined then the mxnet operator can be called directly through SowNode, All optional items can be viewed by ?MXNet` *.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.