# How to get low dimension representation of data based on some knowledge?

I want to map the 500-dims unit vectors to 128-dims and it know some knowledge(43-dims).

So I write this code:

SeedRandom;
INPUTNOTE = 500;
Labels = RandomReal[1, {INPUTNOTE, 43}];
data1 = Transpose[{Range[INPUTNOTE], Labels}];
net1 = NetChain[{128, 256, Ramp, 256, Ramp, 256, Ramp, 43}, "Input" -> INPUTNOTE, "Output" -> 43] generator1 = Function[Map[UnitVector[INPUTNOTE, #[]] -> #[] &,RandomSample[data1, #BatchSize]]];

{net1, LossEvolutionPlot} = NetTrain[net1, generator1, MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, BatchSize ->64, MaxTrainingRounds -> Round[INPUTNOTE/64]*150];


LossEvolutionPlot is: The Maximum MSE is:

Table[Mean[(net1[UnitVector[INPUTNOTE, data1[[i, 1]]]] - data1[[i, 2]])^2], {i, INPUTNOTE}] // Max


4.34154*10^-7

And then extract the encode net:

MatrixPlot[Transpose[NetExtract[net1, 1][UnitVector[INPUTNOTE, #]] & /@ Range[INPUTNOTE]]] But it has some disadvantage,for example

1. if MaxTrainingRounds get larger(MaxTrainingRounds -> Round[INPUTNOTE/64]*300)

LossEvolutionPlot will be strange:(it falls down and then go up) And Maximum MSE is:

Table[Mean[(net1[UnitVector[INPUTNOTE, data1[[i, 1]]]] - data1[[i, 2]])^2], {i, INPUTNOTE}] // Max


0.000100467

2. Another disadvantage is when input have some repeated values,for example:

data2 = Transpose[{RandomChoice[Range, INPUTNOTE], Labels}];
(*same network*)
net2 = NetChain[{128, 256, Ramp, 256, Ramp, 256, Ramp, 43}, "Input" -> INPUTNOTE, "Output" -> 43];
generator2 = Function[Map[UnitVector[INPUTNOTE, #[]] -> #[] &,RandomSample[data2, #BatchSize]]];
{net2, LossEvolutionPlot} = NetTrain[net2, generator2,MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"},BatchSize -> 64, MaxTrainingRounds -> Round[INPUTNOTE/64]*150];


LossEvolutionPlot is: The Maximum MSE is:

Table[Mean[(net2[UnitVector[INPUTNOTE, data2[[i, 1]]]] - data2[[i, 2]])^2], {i, INPUTNOTE}] // Max


0.101637

So is there a better way to reduce dimensions based on some knowledge?

Can we Use embedding layer or using some other methods?

Dimension reduction is often used in text processing applications, in order to collapse the large vocabulary to a smaller manageable size.

You can use the embedding layer in dimension reduction like this:

data = Transpose[{Range[INPUTNOTE], Labels}];
generator = Function[Map[#[] -> #[] &, RandomSample[data1, #BatchSize]]];

net = NetChain[{EmbeddingLayer[128,
"Input" -> NetEncoder[{"Class", Range[1, 500]}]], Ramp,
LinearLayer}]

{net, LossEvolutionPlot} =
NetTrain[net, generator,
MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"},
BatchSize -> 64, MaxTrainingRounds -> 2 Round[INPUTNOTE/64]*300] For your second "disadvantage", in my opinion, I don't think you can do much about it. Allowing the repeated values is similar as encoding the same word into different vectors, which doesn't seem to be consistent to me.