2
$\begingroup$

I want to map the 500-dims unit vectors to 128-dims and it know some knowledge(43-dims).

So I write this code:

SeedRandom[1234];
INPUTNOTE = 500;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data1 = Transpose[{Range[INPUTNOTE], Labels}];
net1 = NetChain[{128, 256, Ramp, 256, Ramp, 256, Ramp, 43}, "Input" -> INPUTNOTE, "Output" -> 43]

enter image description here

generator1 = Function[Map[UnitVector[INPUTNOTE, #[[1]]] -> #[[2]] &,RandomSample[data1, #BatchSize]]];

{net1, LossEvolutionPlot} = NetTrain[net1, generator1, MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, BatchSize ->64, MaxTrainingRounds -> Round[INPUTNOTE/64]*150];

LossEvolutionPlot is:

MSE1

The Maximum MSE is:

Table[Mean[(net1[UnitVector[INPUTNOTE, data1[[i, 1]]]] - data1[[i, 2]])^2], {i, INPUTNOTE}] // Max

4.34154*10^-7

And then extract the encode net:

MatrixPlot[Transpose[NetExtract[net1, 1][UnitVector[INPUTNOTE, #]] & /@ Range[INPUTNOTE]]]

enter image description here

But it has some disadvantage,for example

  1. if MaxTrainingRounds get larger(MaxTrainingRounds -> Round[INPUTNOTE/64]*300)

    LossEvolutionPlot will be strange:(it falls down and then go up)

    enter image description here

    And Maximum MSE is:

    Table[Mean[(net1[UnitVector[INPUTNOTE, data1[[i, 1]]]] - data1[[i, 2]])^2], {i, INPUTNOTE}] // Max
    

    0.000100467

  2. Another disadvantage is when input have some repeated values,for example:

    data2 = Transpose[{RandomChoice[Range[450], INPUTNOTE], Labels}];
    (*same network*)
    net2 = NetChain[{128, 256, Ramp, 256, Ramp, 256, Ramp, 43}, "Input" -> INPUTNOTE, "Output" -> 43];
    generator2 = Function[Map[UnitVector[INPUTNOTE, #[[1]]] -> #[[2]] &,RandomSample[data2, #BatchSize]]];
    {net2, LossEvolutionPlot} = NetTrain[net2, generator2,MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"},BatchSize -> 64, MaxTrainingRounds -> Round[INPUTNOTE/64]*150];
    

    LossEvolutionPlot is:

    enter image description here

    The Maximum MSE is:

    Table[Mean[(net2[UnitVector[INPUTNOTE, data2[[i, 1]]]] - data2[[i, 2]])^2], {i, INPUTNOTE}] // Max
    

    0.101637

So is there a better way to reduce dimensions based on some knowledge?

Can we Use embedding layer or using some other methods?

$\endgroup$
2
$\begingroup$

Dimension reduction is often used in text processing applications, in order to collapse the large vocabulary to a smaller manageable size.

You can use the embedding layer in dimension reduction like this:

data = Transpose[{Range[INPUTNOTE], Labels}];
generator = Function[Map[#[[1]] -> #[[2]] &, RandomSample[data1, #BatchSize]]];

net = NetChain[{EmbeddingLayer[128, 
    "Input" -> NetEncoder[{"Class", Range[1, 500]}]], Ramp, 
   LinearLayer[43]}]

{net, LossEvolutionPlot} = 
 NetTrain[net, generator, 
  MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, 
  BatchSize -> 64, MaxTrainingRounds -> 2 Round[INPUTNOTE/64]*300]

enter image description here

For your second "disadvantage", in my opinion, I don't think you can do much about it. Allowing the repeated values is similar as encoding the same word into different vectors, which doesn't seem to be consistent to me.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.