In the read world, there are many many phones(aa ae ah ao ar aw ax ay b ch d dh ...).
But for brief,I only use three phones(aa ah ae) to illustrate my question.And it only need training Data,no Test Data in all process.
There are two file, one is train_forWolfram.dat(928 KB), the other is phone_id.csv(2 KB).All data are rea-world data,but already normalization.
Importing training file and define the net:
EmbeddingLayerInput is phone ID.INPUT2 is frame information, and the output is audio information in current frame.
data = Partition[BinaryReadList["https://wolfr.am/m6DMp7U1", "Real32"], {1 + 12 + 43}];
generator = Function[<|"EmbeddingLayerInput" -> Rationalize@#[[All, 1]], "Input2" -> #[[All, 2 ;; 13]], "Output" -> #[[All, 14 ;;]]|> &@RandomSample[data, #BatchSize]];
INPUTNOTE = Max[data[[All, 1]]];
net = NetGraph[{EmbeddingLayer[32, "Input" -> NetEncoder[{"Class", Range[0, INPUTNOTE]}]], CatenateLayer[], 64, Ramp, 64, Ramp, 43},
{NetPort["EmbeddingLayerInput"] -> 1 -> 2, NetPort["Input2"] -> 2, 2 -> 3 -> 4 -> 5 -> 6 -> 7},
"Input2" -> 12]
Then train the net(batch size:64 examples, epochs:100 rounds):
{net, LossEvolutionPlot} = NetTrain[net, generator,
MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"},
BatchSize -> 64, MaxTrainingRounds -> Round[Length[data]/64*100]];
View data[[All,1]]:
The adjacent same number means it's the same phone but in different frame.
Also view phone_id.csv(2 KB):
The First row is phone_ID, the second row is phone name.
View the embedding matrix,and try to find whether the same phone has smallest distance.
embeddingWeights = NetExtract[net[[1]], "Weights"];
phoneid = Import["https://wolfr.am/m6DNA4FN", "Table"];
nearstData = Nearest[embeddingWeights, DistanceFunction ->EuclideanDistance];
Assodata = Association@Thread[embeddingWeights ->Range[Length@embeddingWeights]];
ReplaceiablePhone = Table[phoneid[[x]] -> phoneid[[Assodata[nearstData[embeddingWeights[[phoneid[[x, 1]] + 1]], 2][[2]]]]], {x, Length@phoneid}];
MinMax@data[[All, 1]] == MinMax@phoneid[[All, 1]](*True*)
ReplaceiablePhone got these.
It means the phone of ID 0 is much like the phone of ID 110.(correctly),the phone of ID 1 is much like the phone of ID 88.(correctly)...
Then use MatrixPlot to plot these relationships：
confusionMatrixData = #[[1, 2]] -> #[[2, 2]] & /@ ReplaceiablePhone;
phone = Union@confusionMatrixData[[All, 1]];
m = Normal@SparseArray@Normal@Counts[{#[[1]], #[[2]]} & /@ (confusionMatrixData /. Thread[phone -> Range[Length@phone]])];
t = Transpose@Map[Flatten, {#, Reverse@Transpose@#} &[Table[Range[1, 2 # - 1, 2], {#}]] &[Length@m]]/2;
p = MatrixPlot[m, Epilog -> Text @@@ Transpose[{Catenate@m, t}], FrameTicks -> {Transpose@{Range[Length@phone], phone}, Transpose@{Range[Length@phone], Total[m, {1}]}, Transpose@{Range[Length@phone], Total[m, {2}]},Transpose@{Range[Length@phone], phone}}, ImageSize -> 150];
Column[{Row[{Rotate["phone", 90 Degree], p}, Alignment -> Center], "Nearest phone"},
Alignment -> Center]
the ConfusionMatrixPlot of full phone in here
You can see the diagonal of matrix is really bigger than other, and the net really learn something.But it still have misclassified data.
Using TSNE to reduce dimension,we can plot it.
features = DimensionReduce[embeddingWeights, 2, Method -> "TSNE"];
classify = GroupBy[Thread[features -> phoneid[[All, 2]]], Last -> First];
ListPlot[Values[classify], PlotLegends -> PointLegend[97, Keys@classify,
LegendMarkerSize -> 15]]
You can see it has effect in some way ,but could we make it more separable for different phone and more compact for same phone?
The low dimension visualization of full phone is:
If the net has good ability,it will have smallest number of misclassified examples in Confusion Matrix, and the clusters will be more clear,just like this:
So how to make similar objects have similar embedding weights?
Can NetPairEmbeddingOperator help?
