How to get low dimension representation of data based on some knowledge?

Question

I want to map the 500-dims unit vectors to 128-dims and it know some knowledge(43-dims).

So I write this code:

SeedRandom[1234];
INPUTNOTE = 500;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data1 = Transpose[{Range[INPUTNOTE], Labels}];
net1 = NetChain[{128, 256, Ramp, 256, Ramp, 256, Ramp, 43}, "Input" -> INPUTNOTE, "Output" -> 43]

generator1 = Function[Map[UnitVector[INPUTNOTE, #[[1]]] -> #[[2]] &,RandomSample[data1, #BatchSize]]];

{net1, LossEvolutionPlot} = NetTrain[net1, generator1, MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, BatchSize ->64, MaxTrainingRounds -> Round[INPUTNOTE/64]*150];

LossEvolutionPlot is:

The Maximum MSE is:

Table[Mean[(net1[UnitVector[INPUTNOTE, data1[[i, 1]]]] - data1[[i, 2]])^2], {i, INPUTNOTE}] // Max

4.34154*10^-7

And then extract the encode net:

MatrixPlot[Transpose[NetExtract[net1, 1][UnitVector[INPUTNOTE, #]] & /@ Range[INPUTNOTE]]]

But it has some disadvantage,for example

1. if MaxTrainingRounds get larger(MaxTrainingRounds -> Round[INPUTNOTE/64]*300)

   LossEvolutionPlot will be strange:(it falls down and then go up)

   

   And Maximum MSE is:

   Table[Mean[(net1[UnitVector[INPUTNOTE, data1[[i, 1]]]] - data1[[i, 2]])^2], {i, INPUTNOTE}] // Max
   

   0.000100467

2. Another disadvantage is when input have some repeated values,for example:

   data2 = Transpose[{RandomChoice[Range[450], INPUTNOTE], Labels}];
   (*same network*)
   net2 = NetChain[{128, 256, Ramp, 256, Ramp, 256, Ramp, 43}, "Input" -> INPUTNOTE, "Output" -> 43];
   generator2 = Function[Map[UnitVector[INPUTNOTE, #[[1]]] -> #[[2]] &,RandomSample[data2, #BatchSize]]];
   {net2, LossEvolutionPlot} = NetTrain[net2, generator2,MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"},BatchSize -> 64, MaxTrainingRounds -> Round[INPUTNOTE/64]*150];
   

   LossEvolutionPlot is:

   

   The Maximum MSE is:

   Table[Mean[(net2[UnitVector[INPUTNOTE, data2[[i, 1]]]] - data2[[i, 2]])^2], {i, INPUTNOTE}] // Max
   

   0.101637

So is there a better way to reduce dimensions based on some knowledge?

Can we Use embedding layer or using some other methods?

Answer 1

Dimension reduction is often used in text processing applications, in order to collapse the large vocabulary to a smaller manageable size.

You can use the embedding layer in dimension reduction like this:

data = Transpose[{Range[INPUTNOTE], Labels}];
generator = Function[Map[#[[1]] -> #[[2]] &, RandomSample[data1, #BatchSize]]];

net = NetChain[{EmbeddingLayer[128, 
    "Input" -> NetEncoder[{"Class", Range[1, 500]}]], Ramp, 
   LinearLayer[43]}]

{net, LossEvolutionPlot} = 
 NetTrain[net, generator, 
  MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, 
  BatchSize -> 64, MaxTrainingRounds -> 2 Round[INPUTNOTE/64]*300]

For your second "disadvantage", in my opinion, I don't think you can do much about it. Allowing the repeated values is similar as encoding the same word into different vectors, which doesn't seem to be consistent to me.