Here is a (rather poor) example of how you could potentially get started doing this. My apologies for the quality of the answer, I hope someone can come in with a better one. In the meantime it's a way to start.
First we load our network and language model:
sm = NetModel["Deep Speech 2 Trained on Baidu English Data"]
lm = NetModel["Wolfram English Character-Level Language Model V1"]
getFusedProbs[seq_, probs_, lwt_, dwt_, mw_] :=
Module[{lmp = Append[lm[seq, "Probabilities"]^lwt, Missing[] -> mw],
s},
s = Plus @@ KeyIntersection[{
AssociationThread[Keys@probs,
Rescale@Standardize@Values@probs],
AssociationThread[Keys@lmp, Rescale@Standardize@Values@lmp]*lwt
}];
s[StringTake[seq, -1]] = s[StringTake[seq, -1]]*(1 - dwt);
s
]
fusedNet[aud_, lw_: .3, dw_: .5, mw_: .3] := Fold[
Module[{pr = getFusedProbs[#1, #2, lw, dw, mw]}, #1 <>
If[
Or[MissingQ@First@Keys@ReverseSort@pr,
First@Keys@ReverseSort@pr == StringTake[#1, -1] == " "],
"",
First@Keys@ReverseSort@pr
]] &, " ",
AssociationThread[labels, #] & /@
NetReplacePart[sm, "Output" -> None][aud]]
Now we can use fusedNet with various weights:
StringJoin@sm[ExampleData[{"Audio", "FemaleVoice"}]]
"helo helo helo"
fusedNet[ExampleData[{"Audio", "FemaleVoice"}], .8, .1]
"hello hello hello"
Essentially, we multiply the probabilities of the language model by some weight lwt and add them to the probabilities of the speech model, then take the highest result as the most probable character. We fold this across all the raw outputs (that is, without the CTCBeamSearch) of the speech model.
The raw output of the speech model (NetReplacePart[sm, "Output"->Nothing][audio]) is probabilities of each label per sample. However this output doesn't come with labels, so we must add them (hence the AssociationThreads).
We Standardize and Rescale to get both the output of the speech network and the language model on roughly the same scale (nb, this is almost certainly not the right way to do this).
I have included a weight for removing duplicate letters (dwt). This becomes necessary because the speech network's output often has multiple samples for the same letter.
We must append Missing[] to the language model because the speech network output has an extra "nothing" class that isn't in the labels. I have included a default weight for the Missing[] character (mwt) - this is because the language model does not have an extra "nothing" character, which the speech model does have.
Unfortunately, this approach does not work well on your specific example because the typo is at the beginning of the sequence. In general there appears to be a fair bit of tweaking to the weights to get good results.
fusedNet[ExampleData[{"Audio", "MaleVoice"}], .6, .7, .8]
" roht then perfect timeing "
For example, above, the language weight is a little too high, causing the output "timeing", as the language model weights "e" higher than "i" after the sequence "roht then perfect tim".
A better way is I'm sure possible, but my supposition is that I would need to reimplement the CTCBeamSearch decoder and I don't have the will to do that.
Again, my apologies for the quality of the answer, it's quite late at my location. It's quite frustrating that there does not appear to be an example implementation of this in the neural net repo, which really is where this should belong!
