How do I configure the input and output layers of a neural network?

Question

I'm confused about how to set some of the basic properties of a neural network structure using Classify.

I'm trying to set the basic properties of a simple neural network, following Michael Nielsen's online introduction to neural networks and deep learning (which is all I've read on the topic so far, so please level answers accordingly).

I would like to follow the example there, and create a neural network classifier, with three layers of (28*28), 30, and 10 neurons all using logistic sigmoid activation functions. Mathematica seems to hide or override the ability to control some of the necessary properties of the design of the network, and I can't figure out how to take control of them.

For example, if I

T=ExampleData[{"MachineLearning","MNIST"},"TrainingData"];
c=Classify[T,Method->{"NeuralNetwork","HiddenLayers"->{{30,"LogisticSigmoid"}}}]

I get

ClassifierInformation[c]

As far as it goes, these properties are roughly what I'm looking for but it's not clear that they match the network I'm trying to build. Specifically:

1. How do I specify the number of nodes and the activation function for the input layer?
2. How do I specify the mapping between the input examples and the activations of the input layer?
3. How do I specify the number of nodes and the activation function for the output layer?
4. How do I specify mapping between the values of the correct outputs for each input and the activations in the output layer?

Mathematica seems to figure most of this out, recognizing the the input is an "Image", though I see no way of verifying how these are encoded in the input layer or how many nodes are used there. Similarly it seems to sort out the correct number of output categories. Though again, it's unclear how many nodes or used (or how much of the learning process was wasted identifying that number).

The resulting classifier performs is far slower to train than that one constructed by hand.

What options can I use to take control — explicitly setting 1-4 above — of the input and output layers to my neural network?

---

I'm aware that there are some related answers, but they don't address the options I'm looking for.

Answer 1

I think it would be simpler to construct the simple neural network rather than setting the properties in Classify. Here is an example using a three-layer net to classify the MNIST handwritten digits.

We first load the data, and we will take 5000 examples as training data and 1000 examples as test data

resource = ResourceObject["MNIST"];
trainingDataImg = 
  RandomSample[ResourceData[resource, "TrainingData"], 5000];
testDataImg = RandomSample[ResourceData[resource, "TestData"], 1000];

For each training example, we have a grayscale image, and its corresponding digits

In order to train the example, we take the pixel values in the image and flatten them into a list (this is the encode step). And for the digits, we change them into $10\times 1$ vectors, with the corresponding index setting to be 1

trainingData = trainingDataImg /. Rule[x_, y_] :> Flatten[ImageData[x]] -> (If[# == y + 1, 1, 0] & /@ Range[10]);
testData = testDataImg /. Rule[x_, y_] :> Flatten[ImageData[x]] -> (If[# == y + 1, 1, 0] & /@ Range[10]);

Now we can construct the neural network. We will use three layers of (28*28), 30, and 10 neurons. And each neuron will be activated by the sigmoid function:

lenet = NetChain[{
   (*second layer*)
   DotPlusLayer[30],
   ElementwiseLayer[LogisticSigmoid],
   (*third layer*)
   DotPlusLayer[10],
   ElementwiseLayer[LogisticSigmoid]}
  , "Input" -> 784]

which can then be trained

trained = NetTrain[lenet, trainingData,MaxTrainingRounds -> 150]

And we can test the accuracy of this three-layer network

Counts[testData /. Rule[x_, y_] :> First@Flatten[Position[#, Max[#]] &@trained[x]] - 1 == First@Flatten@Position[y, 1] - 1]
(*<|True -> 912, False -> 88|>*)

which is 91.2%.

We can also visualize the hidden layer by extracting their weights:

Partition[
  Image[Partition[#/Max[Abs[#]], 28], ColorSpace -> "Grayscale"] & /@ 
   NetExtract[trained, {1, "Weights"}], 5] // Grid

If we use a single layer with 10 neurons, the weights are easier to make sense of

lenet = NetChain[{DotPlusLayer[10],ElementwiseLayer[LogisticSigmoid]}, "Input" -> 784]
trained = NetTrain[lenet, trainingData, MaxTrainingRounds -> 150]
Grid@Partition[
  MatrixPlot[Partition[#1, 28], 
     ColorFunction -> (Blend[{{0, Blue}, {1/2, Black}, {1, 
           Red}}, #] &), Frame -> None, 
     PlotLabel -> Style[#2, 25]] & @@@ 
   Transpose[{NetExtract[trained, {1, "Weights"}], Range[0, 9]}], 5]