9
$\begingroup$

As a "toy problem" to explore the new neural net library, I wanted to train a neural net with:

Input: 1 real number in the range [0, 1)
Output: the nth digit (i.e. one of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}).

Example:

0.548923584 -> 8
0.678438456 -> 8
0.894258973 -> 4

The following simple NetChain does better than average at classifying the 2nd digit:

digit = 2;

target[x_] := Part[First[RealDigits[x, 10, 6, -1]], digit]

net = 
  NetChain[{100, Ramp, 100, Ramp, 100, Ramp, 100, Ramp, 100, Ramp, 
    100, Ramp, 10, SoftmaxLayer[]}, "Input" -> {1}, 
   "Output" -> NetDecoder[{"Class", Range[0, 9]}]];

trainingData = 
  Table[{x} -> target[x], {x, RandomReal[1, 10^5]}];

trained = NetTrain[net, trainingData];

N[
 Mean[Boole[
   Table[target[x] == trained[{x}], {x, RandomReal[1, 1000]}]]]]

(* 0.55 *)

Very simple variants on the above can be trained to be 90% accurate. But I couldn't get something similar to classify the 3rd digit well.

What's a net design that can do well at classifying the 3rd digit of numbers?

More generally, what are the net design considerations that one should take into account to solve this type of problem?

|improve this question
$\endgroup$
  • $\begingroup$ What I would call the overall "design consideration" is that if you want to classify individual digits of a given number you should make sure that each digit is weighted the same and not considered numerically (e.g in 0.0000000001 the 1 at the end is negligible numerically compared to all the digits before it, so classification on this minute detail is not going to be very effective) $\endgroup$ – Sascha Oct 5 '16 at 8:10
  • $\begingroup$ Consider the following toy task analytically: computing last digit of an integer on range 0 to 99 with only tools you have at your disposal with neural networks. Can you do it without essentially listing all the different cases of tens separately? I think this is not easy unless you have discrete-valued outputs in some parts of the network... $\endgroup$ – kirma Oct 5 '16 at 8:30
  • $\begingroup$ @kirma I am not quite sure if I understood your comment fully but I think the gist of what you said is that one would theoretically need all possibly permutations of n-digit numbers to fully train the network? I don't think this is necessarily the case (and a total coverage of the training data is rare in real life anyway) because the network can generalize on the input it is shown (if not trained to overfit). $\endgroup$ – Sascha Oct 5 '16 at 8:35
  • $\begingroup$ @Sascha Even more extreme case to consider would be the following: input is a two-bit integer (as single input), and output is the least significant bit. What is the simplest network that can solve this problem? How does the network size scale with amount of preceding bits? How do these networks look like? (These are just random thoughts, not an answer!:) $\endgroup$ – kirma Oct 5 '16 at 8:45
  • $\begingroup$ @kirma I don't know about such discrete problems but for any continuous problem there is the Universal approximation theorem. $\endgroup$ – Sascha Oct 5 '16 at 8:53
6
$\begingroup$

An approach is to supply the digits as individual inputs to the network using NetGraph. A possible network (for three input digits e.g. 0.123) could be

net = NetGraph[{
(*Layers*)
Ramp, Ramp, Ramp, CatenateLayer[], DotPlusLayer[100],
DotPlusLayer[100], DotPlusLayer[10], SoftmaxLayer[], 
CrossEntropyLossLayer["Index","Target" -> NetEncoder[{"Class", Range[0, 9]}]]},

(*Layer ordering*)
{NetPort["Digit1"] -> 1, NetPort["Digit2"] -> 2, NetPort["Digit3"] -> 3,
{1, 2, 3} -> 4, 4 -> 5 -> 6 -> 7 -> 8 -> 9, 8 -> NetPort["Output"]},

(*Input and Output specification*) 
"Digit1" -> NetEncoder[{"Class", Range[0, 9], "UnitVector"}],
"Digit2" -> NetEncoder[{"Class", Range[0, 9], "UnitVector"}],
"Digit3" -> NetEncoder[{"Class", Range[0, 9], "UnitVector"}], 
"Output" -> NetDecoder[{"Class", Range[0, 9]}]]

image

Note especially the format of the inputs as one-hot-encoded-vectors (via the "UnitVector"-option of NetEncoder) and the specification of the loss layer. Extending the network above for more input digits (e.g. 0.1234) requires an additional input port per digit that gets connected to the common CatenateLayer.

To generate training data I wrote the helper functions

randomDigits[] := RandomReal[{0, 1}, WorkingPrecision -> 3] // RealDigits // First

toAssociation[targetDigitIndex_][list_] := 
Association["Digit1" -> list[[1]], "Digit2" -> list[[2]], "Digit3" -> list[[3]], 
"Target" -> list[[targetDigitIndex]]]

generateTrainingData[targetDigitIndex_, n_] := 
toAssociation[targetDigitIndex] /@ Table[randomDigits[], n] //Merge[#, Identity] &

The training data format corresponds to the third variant listed in the documentation

documentation

Thus with 

trainingData = generateTrainingData[2, 1000];
trained = NetTrain[net, trainingData, "Loss"]

one gets a trained version of the net above for classifying the second digit (for any other digit just change the first argument to generateTrainingData accordingly). The result can be checked with inputs of the form

trained[<|"Digit1" -> 1, "Digit2" -> 2, "Digit3" -> 3, "Target" -> 5|>, "Output"]
(*yields 2*)

Note that a "Target" has to be supplied (which does however not influence the output). With NetExtract it should be possible to extract only the classification part of the network if so desired (removing the Target and Loss-nodes).

|improve this answer
$\endgroup$
  • $\begingroup$ Impressive. Could you give a reference for learning more about this? $\endgroup$ – Chris Degnen Oct 5 '16 at 8:56
  • $\begingroup$ Do you mean Neural Networks in general or just the implementation? $\endgroup$ – Sascha Oct 5 '16 at 9:01
  • $\begingroup$ @Neural Networks in general which fit with Mathematica's functionality. $\endgroup$ – Chris Degnen Oct 5 '16 at 9:25
  • $\begingroup$ This is a good introduction to the theory of neural networks $\endgroup$ – Sascha Oct 5 '16 at 14:23
  • $\begingroup$ It is very fun when I see this answer after I have learned some neural-networks.I think the result is net = PartLayer[2]..We don't need to train it.If not the parameter will not be changed.. $\endgroup$ – yode Sep 27 '17 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.