5
$\begingroup$

In the read world, there are many many phones(aa ae ah ao ar aw ax ay b ch d dh ...).

But for brief,I only use three phones(aa ah ae) to illustrate my question.And it only need training Data,no Test Data in all process.

There are two file, one is train_forWolfram.dat(928 KB), the other is phone_id.csv(2 KB).All data are rea-world data,but already normalization.

Importing training file and define the net:

EmbeddingLayerInput is phone ID.INPUT2 is frame information, and the output is audio information in current frame.

data = Partition[BinaryReadList["https://wolfr.am/m6DMp7U1", "Real32"], {1 + 12 + 43}];

generator = Function[<|"EmbeddingLayerInput" -> Rationalize@#[[All, 1]], "Input2" -> #[[All, 2 ;; 13]], "Output" -> #[[All, 14 ;;]]|> &@RandomSample[data, #BatchSize]];

INPUTNOTE = Max[data[[All, 1]]];
net = NetGraph[{EmbeddingLayer[32, "Input" -> NetEncoder[{"Class", Range[0, INPUTNOTE]}]], CatenateLayer[], 64, Ramp, 64, Ramp, 43}, 
     {NetPort["EmbeddingLayerInput"] -> 1 -> 2, NetPort["Input2"] -> 2, 2 -> 3 -> 4 -> 5 -> 6 -> 7}, 
     "Input2" -> 12]

Then train the net(batch size:64 examples, epochs:100 rounds):

{net, LossEvolutionPlot} = NetTrain[net, generator, 
   MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, 
   BatchSize -> 64, MaxTrainingRounds -> Round[Length[data]/64*100]]; 

enter image description here

View data[[All,1]]: The adjacent same number means it's the same phone but in different frame.

enter image description here

Also view phone_id.csv(2 KB):

The First row is phone_ID, the second row is phone name.

enter image description here

View the embedding matrix,and try to find whether the same phone has smallest distance.

embeddingWeights = NetExtract[net[[1]], "Weights"];
phoneid = Import["https://wolfr.am/m6DNA4FN", "Table"];
nearstData = Nearest[embeddingWeights, DistanceFunction ->EuclideanDistance];
Assodata = Association@Thread[embeddingWeights ->Range[Length@embeddingWeights]]; 
ReplaceiablePhone = Table[phoneid[[x]] -> phoneid[[Assodata[nearstData[embeddingWeights[[phoneid[[x, 1]] + 1]], 2][[2]]]]], {x, Length@phoneid}];

MinMax@data[[All, 1]] == MinMax@phoneid[[All, 1]](*True*)

ReplaceiablePhone got these.

It means the phone of ID 0 is much like the phone of ID 110.(correctly),the phone of ID 1 is much like the phone of ID 88.(correctly)...

enter image description here

Then use MatrixPlot to plot these relationships:

confusionMatrixData = #[[1, 2]] -> #[[2, 2]] & /@ ReplaceiablePhone;
phone = Union@confusionMatrixData[[All, 1]];
m = Normal@SparseArray@Normal@Counts[{#[[1]], #[[2]]} & /@ (confusionMatrixData /. Thread[phone -> Range[Length@phone]])];
t = Transpose@Map[Flatten, {#, Reverse@Transpose@#} &[Table[Range[1, 2 # - 1, 2], {#}]] &[Length@m]]/2;
p = MatrixPlot[m, Epilog -> Text @@@ Transpose[{Catenate@m, t}], FrameTicks -> {Transpose@{Range[Length@phone], phone}, Transpose@{Range[Length@phone], Total[m, {1}]}, Transpose@{Range[Length@phone], Total[m, {2}]},Transpose@{Range[Length@phone], phone}}, ImageSize -> 150];
Column[{Row[{Rotate["phone", 90 Degree], p}, Alignment -> Center], "Nearest phone"}, 
        Alignment -> Center]

enter image description here

the ConfusionMatrixPlot of full phone in here

You can see the diagonal of matrix is really bigger than other, and the net really learn something.But it still have misclassified data.

Using TSNE to reduce dimension,we can plot it.

features = DimensionReduce[embeddingWeights, 2, Method -> "TSNE"];
classify = GroupBy[Thread[features -> phoneid[[All, 2]]], Last -> First]; 
ListPlot[Values[classify], PlotLegends -> PointLegend[97, Keys@classify, 
         LegendMarkerSize -> 15]]

enter image description here

You can see it has effect in some way ,but could we make it more separable for different phone and more compact for same phone?

The low dimension visualization of full phone is:

enter image description here

If the net has good ability,it will have smallest number of misclassified examples in Confusion Matrix, and the clusters will be more clear,just like this:

enter image description here

So how to make similar objects have similar embedding weights?

Can NetPairEmbeddingOperator help?

$\endgroup$
  • $\begingroup$ hi, dear partida, I happen to see your post, do you have the full labels of your full data? $\endgroup$ – HyperGroups Aug 9 '18 at 9:12
  • $\begingroup$ @HyperGroups what kind of data? for visualize embedding space or confusion matrix? $\endgroup$ – partida Aug 15 '18 at 2:52
3
$\begingroup$

I find a nice way can deal with it.

First thing is the same as before:

data = Partition[BinaryReadList["https://wolfr.am/m6DMp7U1", "Real32"], {1 + 12 + 43}];

generator = Function[<|"EmbeddingLayerInput" -> Rationalize@#[[All, 1]],"Input2" -> #[[All, 2 ;; 13]], "Output" -> #[[All, 14 ;;]]|> &@RandomSample[data, #BatchSize]];

INPUTNOTE = Max[data[[All, 1]]];
net = NetGraph[{EmbeddingLayer[32, "Input" -> NetEncoder[{"Class", Range[0, INPUTNOTE]}]], CatenateLayer[], 64, Ramp, 64, Ramp, 43},{NetPort["EmbeddingLayerInput"] -> 1 -> 2, NetPort["Input2"] -> 2, 2 -> 3 -> 4 -> 5 -> 6 -> 7}, "Input2" -> 12]

{net, LossEvolutionPlot} = NetTrain[net, generator, MeanSquaredLossLayer[], {"TrainedNet", "LossEvolutionPlot"}, BatchSize -> 64, MaxTrainingRounds -> Round[Length[data]/64*100]];

embeddingWeights = NetExtract[net[[1]], "Weights"];
phoneid = Import["https://wolfr.am/m6DNA4FN", "Table"];
nearstData = Nearest[embeddingWeights, DistanceFunction -> EuclideanDistance];
Assodata = Association@Thread[embeddingWeights -> Range[Length@embeddingWeights]];
ReplaceiablePhone = Table[phoneid[[x]] -> phoneid[[Assodata[nearstData[embeddingWeights[[phoneid[[x, 1]] + 1]], 2][[2]]]]], {x, Length@phoneid}];
confusionMatrixData = #[[1, 2]] -> #[[2, 2]] & /@ ReplaceiablePhone;
phone = Union@confusionMatrixData[[All, 1]];
features = DimensionReduce[embeddingWeights, 2, Method -> "TSNE"];
classify = GroupBy[Thread[features -> phoneid[[All, 2]]], Last -> First];
ListPlot[Values[classify], PlotLegends -> PointLegend[97, Keys@classify, LegendMarkerSize -> 15]]

enter image description here

ehh....not well.Still have misclassified data.But we can make it better.

ListPlot[List /@ Median /@ Values[classify], PlotLegends -> PointLegend[97, Keys@classify, LegendMarkerSize -> 15]]
(*get the center of same phone,use median to avoid extreme value*)

enter image description here

phoneidClassify = GroupBy[#[[1]] + 1 -> #[[2]] & /@ phoneid, Last -> First]

<|"aa" -> {1, 4, 5, 8, 10, 12, 13, 18, 19, 21, 27, 31, 32, 34, 35, 37, 45, 47, 49, 50, 51, 55, 57, 59, 60, 62, 63, 67, 69, 70, 73, 74, 75, 78, 91, 95, 98, 100, 105, 110, 111, 114, 116, 121, 133, 135, 137, 140, 145, 146, 154, 159, 161, 164, 167, 172, 174, 176, 177, 180}, "ae" -> {2, 3, 6, 9, 11, 15, 17, 20, 25, 26, 28, 41, 42, 43, 52, 53, 54, 56, 66, 68, 81, 82, 83, 85, 86, 89, 90, 93, 97, 102, 104, 106, 107, 108, 109, 113, 117, 120, 123, 127, 128, 129, 131, 132, 138, 139, 142, 144, 147, 148, 152, 153, 155, 156, 157, 162, 165, 166, 173, 178}, "ah" -> {7, 14, 16, 22, 23, 24, 29, 30, 33, 36, 38, 39, 40, 44, 46, 48, 58, 61, 64, 65, 71, 72, 76, 77, 79, 80, 84, 87, 88, 92, 94, 96, 99, 101, 103, 112, 115, 118, 119, 122, 124, 125, 126, 130, 134, 136, 141, 143, 149, 150, 151, 158, 160, 163, 168, 169, 170, 171, 175, 179}|>

It means the first,4-th, 5-th, 8-th adn so on phone is belong to "aa",the second,third 6-th and so on is belong to "ae"...etc...

Then we use the median vector of the same phone to replace the original vector.

Do[embeddingWeights[[phoneidClassify[i]]] = Median@embeddingWeights[[phoneidClassify[i]]], {i, Keys[phoneidClassify]}];

Now,all the nearest of the phone is the same phone.

So we are train the second net. Same net structure as before.

net2 = NetInitialize@NetGraph[{EmbeddingLayer[32, "Input" -> NetEncoder[{"Class", Range[0, INPUTNOTE]}]], CatenateLayer[], 64, Ramp, 64, Ramp, 43},{NetPort["EmbeddingLayerInput"] -> 1 -> 2, NetPort["Input2"] -> 2, 2 -> 3 -> 4 -> 5 -> 6 -> 7}, "Input2" -> 12];
net2[[1]] = NetReplacePart[net2[[1]], {"Weights" -> embeddingWeights}];

But now ,the most import is,we want the weights of embedding layer change slowly because it has no misclassified data.So the tip is use LearningRateMultipliers.

{net2, LossEvolutionPlot2} = NetTrain[net2,generator,MeanSquaredLossLayer[], 
  {"TrainedNet", "LossEvolutionPlot"}, BatchSize -> 64, MaxTrainingRounds -> Round[Length[data]/64*100], 
  LearningRateMultipliers -> {1 -> 0.0006, _ -> 1}];

enter image description here

Now we plot the confusion matrix

embeddingWeights = NetExtract[net2[[1]], "Weights"];
phoneid = Import["https://wolfr.am/m6DNA4FN", "Table"];
nearstData = Nearest[embeddingWeights, DistanceFunction -> EuclideanDistance];
Assodata = Association@Thread[embeddingWeights -> Range[Length@embeddingWeights]];
ReplaceiablePhone = Table[phoneid[[x]] -> phoneid[[Assodata[nearstData[embeddingWeights[[phoneid[[x, 1]] + 1]], 2][[2]]]]], {x, Length@phoneid}];
confusionMatrixData = #[[1, 2]] -> #[[2, 2]] & /@ ReplaceiablePhone;
phone = Union@confusionMatrixData[[All, 1]];

m = Normal@SparseArray@Normal@Counts[{#[[1]], #[[2]]} & /@ (confusionMatrixData /. Thread[phone -> Range[Length@phone]])];
t = Transpose@Map[Flatten, {#, Reverse@Transpose@#} &[Table[Range[1, 2 # - 1, 2], {#}]] &[Length@m]]/2;
p = MatrixPlot[m, Epilog -> Text @@@ Transpose[{Catenate@m, t}], FrameTicks -> {Transpose@{Range[Length@phone], phone}, Transpose@{Range[Length@phone], Total[m, {1}]}, Transpose@{Range[Length@phone], Total[m, {2}]},Transpose@{Range[Length@phone], phone}}, ImageSize -> 150];
Column[{Row[{Rotate["phone", 90 Degree], p}, Alignment -> Center], "Nearest phone"}, Alignment -> Center]

features = DimensionReduce[embeddingWeights, 2, Method -> "TSNE"];
classify = GroupBy[Thread[features -> phoneid[[All, 2]]], Last -> First];
ListPlot[Values[classify], PlotLegends -> PointLegend[97, Keys@classify, LegendMarkerSize -> 15]]

enter image description here

even on full phone

enter image description here

We see all the phone are classified correctly. And it really has clusters.

---------------------------------------Update 2018/8/15---------------------------------------

I find if there is some aliasing, it comes from the model's structure and database(children's audio books corpus). It's full with emotion that difficult to deal with it.

If pre-trained the model and limit the learning rate, it can not represent the vector of phoneme precisely.

The better way is add additional information to the net's output, we can get more accurate vector to represent the vector of phoneme.

$\endgroup$
  • $\begingroup$ I'm glad you solved your problem. And thanks for sharing the solutions. $\endgroup$ – xslittlegrass Jun 6 '17 at 5:07
  • $\begingroup$ Thank you indeed!!! Giving me so many idea $\endgroup$ – partida Jun 15 '17 at 2:35
2
$\begingroup$

I do not fully understand what you want to do with the two inputs, but here are two examples using NetPairEmbeddingOperator to find an embedding so that the similar objects are close in the embedding. I will use data similar to yours in the first example and some new data in the second example.

Example 1

I use your data but increase the dimension of the data

data = Rule @@@ 
   Transpose[{Flatten[{RandomReal[1, {300, 50}] + 0, 
       RandomReal[0.6, {300, 50}] + 0.4, 
       RandomReal[0.2, {300, 50}] + 0.8}, 1], 
     Flatten[{Table[1, {300}], Table[2, {300}], Table[3, 300]}]}];

generator = 
      Function[Table[
        With[{sp = RandomSample[data, 2]}, 
         sp[[All, 1]] -> Not[Equal @@ sp[[All, 2]]]], {#BatchSize}]];

We first define the embedding network

embnet = NetChain[{
   LinearLayer[100], Ramp, LinearLayer[2]},
  "Input" -> {50}];

Then construct a loss network to measure the performance of the embedding network using NetPairEmbeddingOperator

net = NetPairEmbeddingOperator[embnet];

Train this network, and extract the trained embedding network

trained = NetTrain[net, generator];
embnetTrained = NetExtract[trained, "Net"];

We can generate the embedded values of the original data, and we see that the three classes has been clearly separated.

color = {Blue, Green, Red};
Graphics[{PointSize[Medium], color[[#[[2]]]], Point[embnetTrained[#[[1]]]]} & /@
   data, AspectRatio -> 1, PlotRange -> All]

enter image description here

Example 2

In this example, I'm trying to train an embedding network to separate data generated from three different Gaussian distributions.

data1 = With[{μ = 0.5, σ = 0.4}, 
   Table[Table[
      Exp[-((x - μ)^2./(2. σ^2))] + 
       0.4 RandomReal[], {x, -1, 1, 0.01}] -> 1, {100}]];
data2 = With[{μ = 0., σ = 0.4}, 
   Table[Table[
      Exp[-((x - μ)^2./(2. σ^2))] + 
       0.4 RandomReal[], {x, -1, 1, 0.01}] -> 2, {100}]];
data3 = With[{μ = -0.5, σ = 0.4}, 
   Table[Table[
      Exp[-((x - μ)^2./(2. σ^2))] + 
       0.4 RandomReal[], {x, -1, 1, 0.01}] -> 3, {100}]];


data = RandomSample[Chop@Join[data1, data2, data3]];

ListPlot[data[[All, 1]], Joined -> True, PlotRange -> All, 
 PlotStyle -> ColorData[97, "ColorList"][[data[[All, 2]]]]]

enter image description here

We again define an embedding network and a loss network

generator = 
  Function[Table[
    With[{sp = RandomSample[data, 2]}, 
     sp[[All, 1]] -> (! Equal @@ sp[[All, 2]])], {#BatchSize}]];
embnet = NetChain[{LinearLayer[100], Ramp, 10, Ramp, LinearLayer[2]}, 
  "Input" -> {201}]
net = NetPairEmbeddingOperator[embnet]

Train the loss network and extract the trained embedding network

trained = NetTrain[net, generator, MaxTrainingRounds -> 100]
embnetTrained = NetExtract[trained, "Net"];

Legended[Graphics[{PointSize[Medium], color[[#[[2]]]], 
     Point[embnetTrained[#[[1]]]]} & /@ data, AspectRatio -> 1], 
 PointLegend[color, {1, 2, 3}]]

We again see a clean separation for the three different groupsenter image description here

$\endgroup$
  • $\begingroup$ Thank you very much..But because I describe the question not clear before,I add more information. In your answer,I don't think use NetChain[...,Ramp,...] to replace EmbeddingLayer is a good idea.Because every row of weights of EmbeddingLayer is a 32-dims feature that can describe the phone. $\endgroup$ – partida Jun 2 '17 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.