Similar data,same net,different level of MSE loss?

Question

Using same net and similar data,but MSE loss is very different.

This is the original data,and it can not learn very well.

originalData = Get["https://wolfr.am/lRWRvwBp"];
originalGenerator = <|"EmbeddingLayerInput" -> Rationalize@#[[All, 1]], "Input2" -> #[[All, 2 ;; 13]], "Output" -> #[[All, 14 ;;]]|> &@RandomSample[originalData, #BatchSize] &;

originalGenerator[<|"BatchSize" -> 3|>]

net = NetGraph[{EmbeddingLayer[128, "Input" -> NetEncoder[{"Class", Range[0, Max@data[[All, 1]]]}]], 
     CatenateLayer[], 512, Ramp, 43},
     {NetPort["EmbeddingLayerInput"] -> 1 -> 2, NetPort["Input2"] -> 2 -> 3 -> 4 -> 5},
     "Input2" -> 12]

NetTrain[net, originalGenerator, MeanSquaredLossLayer[],
 "LossEvolutionPlot", BatchSize -> 300, MaxTrainingRounds -> 200]

And this is test data,it learns well.

INPUTNOTE = Length[originalData];
testData = Transpose[{Join[List /@ originalData[[All, 1]], RandomReal[1, {INPUTNOTE, 12}], 2], RandomReal[1, {INPUTNOTE, 43}]}];
testGenerator = <|"EmbeddingLayerInput" -> Rationalize@#[[All, 1, 1]],"Input2" -> #[[All, 1, 2 ;;]], "Output" -> #[[All, 2]]|> &@RandomSample[testData, #BatchSize] &;

testGenerator[<|"BatchSize" -> 3|>]

NetTrain[net, testGenerator, MeanSquaredLossLayer[], 
  "LossEvolutionPlot", BatchSize -> 300, MaxTrainingRounds -> 200]

So why using similar data but different level of MES loss?

I guess weather the scale of different dimensions is too big,so I try originalData[[All, 2 ;;]] = Standardize /@ originalData[[All, 2 ;;]];,but not work.

And how to pre-process data to get much smaller MSE loss?

Ps: I draw the histogram of every dimension of data,yellow is original data,red is test data:

originalHist = Histogram[#, 10, Ticks -> None] & /@ Transpose[originalData];
testHist = Histogram[#, 10, Ticks -> None] & /@ Transpose[Flatten /@ testData];
GraphicsGrid[Partition[Flatten@Transpose@{originalHist, testHist}, 8],Frame -> All, Background -> {{{LightYellow, LightRed}}, None}]

View the histogram plot can find the histogram of test data is more stable.Is this the reason why second example better than first?

Answer 1

If we plot the distribution of your training and testing data, we can see they are very different:

originalData = Get["https://wolfr.am/lRWRvwBp"];
originalData[[All, 2 ;;]] = Standardize /@ originalData[[All, 2 ;;]];

Histogram[{Flatten[originalData[[All, 2 ;;]]], 
  Flatten[{testData[[All, 1, 2 ;;]], testData[[All, 2]]}]}, 
 PlotRange -> All]

We see that even after you standardize the training data, there is still a large portion of them outside of the range of 1. This sometimes will make the convergence slow, since some of the ReLU units will be saturated or dead and the gradient will not flow in the network.

This can be solved by shifting the data into the region of 1:

originalData = Get["https://wolfr.am/lRWRvwBp"];
originalData[[All, 2 ;;]] = 
  0.5 + originalData[[All, 2 ;;]]/Max[originalData[[All, 2 ;;]]];

Now the training converges much faster