# How to ensure the iteration will converge when train the NetWork

Let's generate some points for test

disk1 = Disk[{0, 0}, 1, {0, Pi/4}];
disk2 = Disk[{0, 0}, 1, {Pi/4, 3 Pi/4}];
disk3 = Disk[{0, 0}, 1, {3 Pi/4, 5 Pi/4}];
disk4 = Disk[{0, 0}, 1, {5 Pi/4, 2 Pi}];
pts = RandomPoint[#, RandomInteger[{30, 40}]] & /@ {disk1, disk2,
disk3, disk4};
Graphics[{PointSize[0.02],
Flatten@MapAt[Point,
Flatten[{{Red, Green, Blue, Black}, pts}, {{2}, {1}}], {All, 2}]}]


This is the train sets

data = RandomSample[Flatten[Thread /@ Thread[pts -> Range[4]]]];


This is a net I build to classify those points

net = NetGraph[{4, Ramp,
SoftmaxLayer[
"Output" -> NetDecoder[{"Class", {1, 2, 3, 4}}]]}, {1 -> 2 -> 3},
"Input" -> 2]


Let's trained it

trainedNet =
NetTrain[net, data, ValidationSet -> Scaled[.2],
TargetDevice -> "GPU"]


As the Training Progress, I will should have a good result.But actually not..

ContourPlot[trainedNet[{x, y}], {x, -1, 1}, {y, -1, 1},
Epilog -> {PointSize[0.02],
Flatten@MapAt[Point,
Flatten[{{Red, Green, Blue, Black}, pts}, {{2}, {1}}], {All,2}]}]


As you see, I get a very bad trained NetWork

But,because of the initial values is random value, sometimes I can get a very good result like

Of course we always hope to get a good trained result. Can anyone can give some advice?

I think the problem is because of the Ramp before the SoftmaxLayer. Usually, we don't use activation function for the last linear layer. After removing the Ramp, your network will be more stable. I get 10 out of 10 correct clusterings after this modification.
net = NetGraph[{4,