35
$\begingroup$

Is there a way to create our own custom layers for Neural Networks in Mathematica ? I suspect it should be possible, any pointers or directions on how to go about it?

I am interested in functionality like the "Lambda Layer" in Keras API.

$\endgroup$
6
  • 11
    $\begingroup$ I would love to see some docs on NeuralNetworks`DefineLayer $\endgroup$
    – swish
    Feb 20, 2017 at 20:18
  • $\begingroup$ Thats interesting ... $\endgroup$ Feb 20, 2017 at 20:23
  • 1
    $\begingroup$ @swish Can you give a example that success to use it? $\endgroup$
    – yode
    Feb 21, 2017 at 2:17
  • $\begingroup$ @yode Don't have one, that is the problem. $\endgroup$
    – swish
    Feb 21, 2017 at 10:07
  • $\begingroup$ I am looking for a functionality that makes a layer can also let its weights be part of the output, so that one can use them to define a more sophisticated loss function, e.g., imposing constraints on the weights including but not limited to sum of absolute squares. $\endgroup$ Mar 28, 2018 at 6:18

4 Answers 4

16
$\begingroup$

Supporting custom layers is on our to-do list, and should be ready for either 11.2 or 11.3.

For interest: what applications do you want custom layers for? And how performant do you need your custom layer to be? (for example, do you want to be able to write your own CUDA implementation? Or are you happy with writing your own CPU layer?)

$\endgroup$
13
  • 1
    $\begingroup$ Not a custom layer per se, but the ability to costomize weight sharing would be really useful. $\endgroup$
    – Sascha
    Mar 26, 2017 at 20:12
  • 2
    $\begingroup$ @AlexeyGolyshev: Mathematica is designed to be a functional language where things don't usually have state. Our network framework is built on the same principle. This means that we expose the RNN states explicitly as arguments (see the docs for LSTM etc). Its definitely annoying to use this for generation from RNN's. Expect major useability improvements for generation in 11.2. Also phased-lstm: will take a look. Can you implement it in 11.1 using NetFoldOperator? $\endgroup$
    – Sebastian
    Mar 28, 2017 at 10:15
  • 3
    $\begingroup$ Sebastian, thanks for your response. I am using Mathematica 11.3 and am wondering if this functionality has been actually implemented. One reason to use it is to be able to take advantage of the rest of the system while investigating your own ideas in the NN area. By the way, thanks for the great article on O'Reilly! $\endgroup$
    – LevK
    Mar 28, 2018 at 19:34
  • 5
    $\begingroup$ @levK: I implemented a general custom-layer mechanism some months ago, but it turned out to cause many problems due to the asynchronous nature of the callbacks. For 12, a really good solution should exist though: custom layers via using the upcoming compiler infrastructure, with the end-game being direct compilation to CUDA! So stay tuned :-) $\endgroup$
    – Sebastian
    Apr 4, 2018 at 8:48
  • 3
    $\begingroup$ @Sebastian did this ever happen? $\endgroup$
    – user5601
    Sep 12, 2019 at 5:22
10
+50
$\begingroup$

Here is a practical example using NeuralNetworks`DefineLayer: ModelZoo/PGGAN-128 trained on Anime

PixelNorm.m defines a newPixelNormalizationLayer:

Input: ChannelT[$$Channels, TensorT[$$InputDimensions]]
Output: ChannelT[$$Channels, TensorT[$$InputDimensions]]

Parameters:
    $Epsilon: Defaulting[ScalarT, 10^-8]
    $$Channels: SizeT
    $$InputDimensions: SizeListT[SizeT]

Writer: Function[
    input = GetInput["Input", "Batchwise"];
    path = SowNode["mean", SowSquare@input, "axis" -> 1, "keepdims" -> True];
    path = SowRSqrt@SowNode["_PlusScalar", path, "scalar" -> #Epsilon];
    output = SowNode["broadcast_mul", {input, path}];
    SetOutput["Output", output]
]

Suffix: "alizationLayer"

enter image description here

The executable notebook can be downloaded from PGGAN-128.trained.on.Anime.zip

Every time you load the network, you must first load the custom layers through NeuralNetworks`DefineLayer.

Such a network cannot be submitted to NeuralNetRepository, and currently there is no solution similar to FunctionRepository.


You can see that the custom layer uses a domain-specific language

All options can be viewed by NeuralNetworks`Private`DefineLayer`LayerDefinitionT[[1, All, 1]]

There also contains type and interface information.


These options have their own roles, but there are only two necessary parts of a layer: Input/Output and Writer.

If you don't want to check the input and output, you can directly set it to AnyTensorT, which will not do any shape check.

Writer can specify how to convert to sow node.

Sow is a number of predefined operators.You can view all definitions via ? NeuralNetworks`Private`Sow*.

If it is not defined then the mxnet operator can be called directly through SowNode, All optional items can be viewed by ?MXNet` *.

$\endgroup$
9
$\begingroup$

There are some documents written in markdown, under the path SystemFiles\Components\NeuralNetworks. Those under Layers are particularly of interest to question in OP.

FileNameJoin[{$InstallationDirectory
       , "SystemFiles", "Components", "NeuralNetworks"
       }] // FileNames["*.md", #, ∞] &

(*
SystemFiles\Components\NeuralNetworks\Encoders\EncoderFunctionAPI.md
                         .  .  .     \Layers\LayerDefinitions.md
                         .  .  .     \Layers\ShapePolymorphism.md
                         .  .  .     \Layers\Writers.md
                         .  .  .     \MXNet\Evaluation.md
                         .  .  .     \NetTrain.m\InternalOptions.md
                         .  .  .     \NetTrain.m\Overview.md
                         .  .  .     \Organization.md
*)
$\endgroup$
3
  • 1
    $\begingroup$ Too bad that those source code are no longer accessible in 12.3. $\endgroup$ Jul 9, 2021 at 18:15
  • $\begingroup$ @xslittlegrass Mine are still there, under .....\AppData\Roaming\Mathematica\Paclets\Repository\NeuralNetworks-12.0.11. There are multiple versions of NeuralNetworks paclets in that folder. $\endgroup$
    – Silvia
    Jul 9, 2021 at 18:25
  • $\begingroup$ I only see a NeuralNetworks.mx file in that folder, the same as that in the SystemFiles\Components\NeuralNetworks folder. $\endgroup$ Jul 9, 2021 at 18:38
7
$\begingroup$

Here's my guess for how to call DefineLayer based on the DownValues and some minimal spelunking.

First we define an Association defining the layer à la NeuralNetworks`$LayerData (seems that you need to drop {"Symbol", "Type"} based on how I was calling it):

fakeAssoc=
 KeyDrop[ NeuralNetworks`$LayerData["Linear"], {"Symbol", "Type"}]

Then just give it a name as a String:

NeuralNetworks`DefineLayer["Fake", fakeAssoc];

Now NeuralNetworks`FakeLayer is defined:

NeuralNetworks`FakeLayer[{}]

fake layer

If we want a non "NeuralNetworks`" level symbol instead, we'd need to make sure the NameQ["System`"<>name] is `True:

System`FakeLayer;
NeuralNetworks`DefineLayer["Fake",
  KeyDrop[ NeuralNetworks`$LayerData["Linear"], {"Symbol", "Type"}]
  ];

FakeLayer[{}]

fake 2

I think NeuralNetworks`$LayerData["Linear"] should be clear enough as a template for doing this. The parameter names seem pretty straight-forward:

NeuralNetworks`$LayerData["Linear"] // Keys

{"Inputs", "Outputs", "States", "Arrays", "Parameters", \
"ParameterCoercions", "ParameterDefaults", "InferenceRules", \
"RuntimeInferenceRules", "ShapeFunction", "RankFunction", \
"ExtraShapeFunctionTensors", "PostInferenceFunction", \
"PostConstructionFunction", "AuxArrays", "Writer", "Upgraders", \
"MXNet", "SubNets", "IsLoss", "IsOperator", "IsMultiport", \
"InheritsFrom", "Suffix", "SummaryFunction", "Tests", \
"AllowDynamicDimensions", "ReshapeParams", "ArgumentRewriter", \
"CUDNNArray", "Constraints", "StateExpanding", "MaxArgCount", \
"MinArgCount", "PosArgCount", "PosArgs", "SourceFile", "Type", \
"Symbol"}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.