4
$\begingroup$
SeedRandom[0];
X = RandomInteger[{1, 5000}, {100000, 50}];
Y = RandomInteger[{0, 1}, 100000];

For net1 and net2 errors are around 50% for train and validation. No questions. Garbage in, garbage out.

net3 has error on train 24.2%. This is good. But why EmbeddingLayer improves the quality of model in compare to net2? How to achieve this error on validation? Right now error on validation is 50.4% and logloss is 1.41.

I have anonymized dataset. I don't know meaning of features. Data has type "Real" and there are a lot of duplicates. I found that I can get good result during training if replace data with integer indices (using ArrayComponents) and use EmbeddingLayer. But result on validation is bad. I decided to check this neural network architecture on random data. And it can overfit random data!

How to explain overfitting?

net1 = NetChain[
  {
   128,
   Ramp,
   128,
   2,
   SoftmaxLayer[]
   },
  "Input" -> 50,
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

enter image description here

NetTrain[
 net1,
 N@X[[;; 70000]] -> Y[[;; 70000]],
 All,
 ValidationSet -> X[[70001 ;;]] -> Y[[70001 ;;]],
 MaxTrainingRounds -> 10,
 TargetDevice -> "GPU"
 ]

enter image description here

net2 = NetChain[
  {
   ReshapeLayer[{50, 1}],
   LongShortTermMemoryLayer[128],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Input" -> 50,
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

enter image description here

NetTrain[
 net2,
 N@X[[;; 70000]] -> Y[[;; 70000]],
 All,
 ValidationSet -> X[[70001 ;;]] -> Y[[70001 ;;]],
 MaxTrainingRounds -> 10,
 TargetDevice -> "GPU"
 ]

enter image description here

net3 = NetChain[
  {
   EmbeddingLayer[10, 5000],
   LongShortTermMemoryLayer[128],
   SequenceLastLayer[],
   DropoutLayer[.2],
   2,
   SoftmaxLayer[]
   },
  "Input" -> 50,
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

enter image description here

NetTrain[
 net3,
 X[[;; 70000]] -> Y[[;; 70000]],
 All,
 ValidationSet -> X[[70001 ;;]] -> Y[[70001 ;;]],
 MaxTrainingRounds -> 10,
 TargetDevice -> "GPU"
 ]

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.