There is only one NetModel for speech recognition. As is stated in the network repository, models trained with CTC loss have difficulties in producing correct spelling:

StringJoin @ NetModel["Deep Speech 2 Trained on Baidu English Data"][
  ExampleData[{"Audio", "MaleVoice"}]]

enter image description here

The docs suggest incorporating a language model into the beam search decoding, but I have no idea

  1. Where to find a "language model" in NetGraph usable form
  2. How to incorporate this with a NetDecoder

Has anyone tried this and/or are there any examples or blog posts to work off of here?

  • $\begingroup$ Here is an English language model: resources.wolframcloud.com/NeuralNetRepository/resources/… $\endgroup$
    – Carl Lange
    Commented Mar 26, 2019 at 19:14
  • $\begingroup$ Nice, half way home! $\endgroup$
    – M.R.
    Commented Mar 27, 2019 at 4:28
  • $\begingroup$ There is a technique called shallow fusion that is used to do this. However, we have a few problems implementing this approach, not least that the language model and the beam search decoder have different alphabets. $\endgroup$
    – Carl Lange
    Commented Mar 27, 2019 at 8:39

1 Answer 1


Here is a (rather poor) example of how you could potentially get started doing this. My apologies for the quality of the answer, I hope someone can come in with a better one. In the meantime it's a way to start.

First we load our network and language model:

sm = NetModel["Deep Speech 2 Trained on Baidu English Data"]
lm = NetModel["Wolfram English Character-Level Language Model V1"]

getFusedProbs[seq_, probs_, lwt_, dwt_, mw_] :=
 Module[{lmp = Append[lm[seq, "Probabilities"]^lwt, Missing[] -> mw], 
  s = Plus @@ KeyIntersection[{
      AssociationThread[Keys@lmp, Rescale@Standardize@Values@lmp]*lwt
  s[StringTake[seq, -1]] = s[StringTake[seq, -1]]*(1 - dwt);

fusedNet[aud_, lw_: .3, dw_: .5, mw_: .3] := Fold[
  Module[{pr = getFusedProbs[#1, #2, lw, dw, mw]}, #1 <>
       First@Keys@ReverseSort@pr == StringTake[#1, -1] == " "],
      ]] &, " ", 
  AssociationThread[labels, #] & /@ 
   NetReplacePart[sm, "Output" -> None][aud]]

Now we can use fusedNet with various weights:

StringJoin@sm[ExampleData[{"Audio", "FemaleVoice"}]]

"helo helo helo"

fusedNet[ExampleData[{"Audio", "FemaleVoice"}], .8, .1]

"hello hello hello"

Essentially, we multiply the probabilities of the language model by some weight lwt and add them to the probabilities of the speech model, then take the highest result as the most probable character. We fold this across all the raw outputs (that is, without the CTCBeamSearch) of the speech model.

The raw output of the speech model (NetReplacePart[sm, "Output"->Nothing][audio]) is probabilities of each label per sample. However this output doesn't come with labels, so we must add them (hence the AssociationThreads).

We Standardize and Rescale to get both the output of the speech network and the language model on roughly the same scale (nb, this is almost certainly not the right way to do this).

I have included a weight for removing duplicate letters (dwt). This becomes necessary because the speech network's output often has multiple samples for the same letter.

We must append Missing[] to the language model because the speech network output has an extra "nothing" class that isn't in the labels. I have included a default weight for the Missing[] character (mwt) - this is because the language model does not have an extra "nothing" character, which the speech model does have.

Unfortunately, this approach does not work well on your specific example because the typo is at the beginning of the sequence. In general there appears to be a fair bit of tweaking to the weights to get good results.

fusedNet[ExampleData[{"Audio", "MaleVoice"}], .6, .7, .8]

" roht then perfect timeing "

For example, above, the language weight is a little too high, causing the output "timeing", as the language model weights "e" higher than "i" after the sequence "roht then perfect tim".

A better way is I'm sure possible, but my supposition is that I would need to reimplement the CTCBeamSearch decoder and I don't have the will to do that.

Again, my apologies for the quality of the answer, it's quite late at my location. It's quite frustrating that there does not appear to be an example implementation of this in the neural net repo, which really is where this should belong!

  • $\begingroup$ This is a great start! Is there any specific paper or blogs you can add as a reference? $\endgroup$
    – M.R.
    Commented Mar 28, 2019 at 15:39
  • $\begingroup$ Not really - this is essentially based on the shallow fusion approach laid out here, but I stress that it's not a proper implementation of it. Searching Arxiv for shallow fusion should get you far :) $\endgroup$
    – Carl Lange
    Commented Mar 28, 2019 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.