1
$\begingroup$

I would like to visualize and cluster vectors learned in the latent space of a Neural Network.

I am trying to cluster audio tokens, but I don't know what they represent. So the task is to automatically label all tokens that are similar.

I am trying to explore Mathematica, instead of python, and learning from this tutorial: https://www.wolfram.com/language/12/machine-learning-for-audio/extract-features-using-a-neural-net.html?product=mathematica

Consider this:

(* generate random audio token for sake of example *)
audioProps = 
  RandomChoice[{"Sin", "Triangle", "Sawtooth", "Square", "Pulse", 
    "Impulse"}, 1000];
audioFreqs = Table[RandomInteger[{220, 880}], 1000];
audio = AudioGenerator /@ Transpose[{audioProps, audioFreqs}];

now, from the tutorial:

net = NetModel["Wolfram AudioIdentify V1 Trained on AudioSet Data", 
  "Size" -> "Small"]
mainNet = NetExtract[net, {1, "Net"}]
featureExtractor = 
 NetChain[{NetMapOperator[NetDrop[mainNet, -3]], 
   AggregationLayer[Max, 1], FlattenLayer[]}, 
  "Input" -> net[["Input"]]]

At this point I do:

fsp = FeatureSpacePlot[# & /@ audio, FeatureExtractor -> featureExtractor]

But what I want to do is to extract the latent space, and have more control. Example, I would like to cluster the coordinates shown on FeatureSpacePlot, or even better cluster the vectors from the last layer learned by featureExtractor.

I tried:

NetExtract["featureExtractor", 3] // last layer

and

NetExtract["fsp", Last] // latent space ??

Both give an error.

$\endgroup$
1
  • $\begingroup$ Your NetExtract commands are wrong because you've got quotes "..." which you don't need. Also // is not a comment in Mathematica and will be interpreted as Postfix. These (* ... *) and these (** ... **) are for comments. $\endgroup$
    – flinty
    Apr 25, 2023 at 14:31

1 Answer 1

1
$\begingroup$

You can use DimensionReduce like this:

drvecs = 
  DimensionReduce[audio, 2, FeatureExtractor -> featureExtractor, 
   Method -> "UMAP"];
clusters = FindClusters[drvecs];
ListPlot[clusters, AspectRatio -> 1]

enter image description here

$\endgroup$
1
  • $\begingroup$ thank you @flinty, sorry for having oversught your answer $\endgroup$
    – user305883
    Jun 30, 2023 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.