6
$\begingroup$

As this playground show after you click this button, just four levels can solve the xor problem. So I try to simulate it in Mathematica

Generate test points

    disk1 = Disk[{0, 0}, 1, {0, Pi/2}];
    disk2 = Disk[{0, 0}, 1, {Pi/2, Pi}];
    disk3 = Disk[{0, 0}, 1, {Pi, 3 Pi/2}];
    disk4 = Disk[{0, 0}, 1, {3 Pi/2, 2 Pi}];
    pts = RandomPoint[#, RandomInteger[{30, 40}]] & /@ {disk1, disk2, 
    disk3, disk4};
    Graphics[{PointSize[0.02], {Red, Point[Catenate[pts[[{1, 3}]]]], Blue,
    Point[Catenate[pts[[{2, 4}]]]]}}]

Mathematica graphics

Training the network

data = Flatten[
   Thread /@ {Catenate[pts[[{1, 3}]]] -> 1, 
     Catenate[pts[[{2, 4}]]] -> 0}];
net = NetChain[{4, Ramp, 2, Ramp, 
    SoftmaxLayer["Output" -> NetDecoder[{"Class", {0, 1}}]]}, 
   "Input" -> 2];
trainednet = 
 NetTrain[net, data, ValidationSet -> Scaled[.2], 
  TargetDevice -> "GPU"]

Show the trained result

ContourPlot[trainednet[{x, y}], {x, -1, 1}, {y, -1, 1}, 
 Epilog -> {PointSize[0.02], Red, Point[Catenate[pts[[{1, 3}]]]], 
   Blue, Point[Catenate[pts[[{2, 4}]]]]}]

We get a good result as the above image, but actually,we often get a frustrated result like following

I'm confused, because the playground always converge well. Why I use same network layer but cannot get same good result? How to improve it?

$\endgroup$
3
  • $\begingroup$ Without testing anything: The Ramp before the SoftmaxLayer looks suspicious $\endgroup$ Commented Oct 10, 2017 at 15:11
  • $\begingroup$ @nikie Then. :) And as my textbook,that is not a problem. $\endgroup$
    – yode
    Commented Oct 10, 2017 at 15:15
  • $\begingroup$ @nikie It will be better a little indeed $\endgroup$
    – yode
    Commented Oct 10, 2017 at 15:57

1 Answer 1

3
$\begingroup$

(Sidenote: It's annoying if you change the question so the answer you got doesn't work anymore...)

I think this is normal for ReLU activation. If all channels in the middle layer are 0 in one quadrant at initialization, the network will never learn in that quadrant, because the gradient is 0. It happens in the "playground", too. (Less frequently though - maybe they have a different initialization strategy.)

How to improve it?

Either use more channels, or use an activation function that has a nonzero gradient everywhere (like Tanh)

$\endgroup$
1
  • $\begingroup$ If you mean I change the number from 10 to 4, because that is a typo. I want to simulate playground and it just need 4 levels. $\endgroup$
    – yode
    Commented Oct 10, 2017 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.