How can I train a neural net to decide the 3rd digit of a number?

Question

As a "toy problem" to explore the new neural net library, I wanted to train a neural net with:

Input: 1 real number in the range [0, 1)
Output: the nth digit (i.e. one of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}).

Example:

0.548923584 -> 8
0.678438456 -> 8
0.894258973 -> 4

The following simple NetChain does better than average at classifying the 2nd digit:

digit = 2;

target[x_] := Part[First[RealDigits[x, 10, 6, -1]], digit]

net = 
  NetChain[{100, Ramp, 100, Ramp, 100, Ramp, 100, Ramp, 100, Ramp, 
    100, Ramp, 10, SoftmaxLayer[]}, "Input" -> {1}, 
   "Output" -> NetDecoder[{"Class", Range[0, 9]}]];

trainingData = 
  Table[{x} -> target[x], {x, RandomReal[1, 10^5]}];

trained = NetTrain[net, trainingData];

N[
 Mean[Boole[
   Table[target[x] == trained[{x}], {x, RandomReal[1, 1000]}]]]]

(* 0.55 *)

Very simple variants on the above can be trained to be 90% accurate. But I couldn't get something similar to classify the 3rd digit well.

What's a net design that can do well at classifying the 3rd digit of numbers?

More generally, what are the net design considerations that one should take into account to solve this type of problem?

Answer 1

An approach is to supply the digits as individual inputs to the network using NetGraph. A possible network (for three input digits e.g. 0.123) could be

net = NetGraph[{
(*Layers*)
Ramp, Ramp, Ramp, CatenateLayer[], DotPlusLayer[100],
DotPlusLayer[100], DotPlusLayer[10], SoftmaxLayer[], 
CrossEntropyLossLayer["Index","Target" -> NetEncoder[{"Class", Range[0, 9]}]]},

(*Layer ordering*)
{NetPort["Digit1"] -> 1, NetPort["Digit2"] -> 2, NetPort["Digit3"] -> 3,
{1, 2, 3} -> 4, 4 -> 5 -> 6 -> 7 -> 8 -> 9, 8 -> NetPort["Output"]},

(*Input and Output specification*) 
"Digit1" -> NetEncoder[{"Class", Range[0, 9], "UnitVector"}],
"Digit2" -> NetEncoder[{"Class", Range[0, 9], "UnitVector"}],
"Digit3" -> NetEncoder[{"Class", Range[0, 9], "UnitVector"}], 
"Output" -> NetDecoder[{"Class", Range[0, 9]}]]

Note especially the format of the inputs as one-hot-encoded-vectors (via the "UnitVector"-option of NetEncoder) and the specification of the loss layer. Extending the network above for more input digits (e.g. 0.1234) requires an additional input port per digit that gets connected to the common CatenateLayer.

To generate training data I wrote the helper functions

randomDigits[] := RandomReal[{0, 1}, WorkingPrecision -> 3] // RealDigits // First

toAssociation[targetDigitIndex_][list_] := 
Association["Digit1" -> list[[1]], "Digit2" -> list[[2]], "Digit3" -> list[[3]], 
"Target" -> list[[targetDigitIndex]]]

generateTrainingData[targetDigitIndex_, n_] := 
toAssociation[targetDigitIndex] /@ Table[randomDigits[], n] //Merge[#, Identity] &

The training data format corresponds to the third variant listed in the documentation

Thus with

trainingData = generateTrainingData[2, 1000];
trained = NetTrain[net, trainingData, "Loss"]

one gets a trained version of the net above for classifying the second digit (for any other digit just change the first argument to generateTrainingData accordingly). The result can be checked with inputs of the form

trained[<|"Digit1" -> 1, "Digit2" -> 2, "Digit3" -> 3, "Target" -> 5|>, "Output"]
(*yields 2*)

Note that a "Target" has to be supplied (which does however not influence the output). With NetExtract it should be possible to extract only the classification part of the network if so desired (removing the Target and Loss-nodes).