I have a text and vocabulary (text has two "the"
entries):
list = {"the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"}
voc = {"the", "quick", "brown", "fox", "jumps", "over", "lazy", "dog"}
And I can do integer codes with NetEncoder
.
NetEncoder[{"Tokens", voc}][list]
{{1}, {2}, {3}, {4}, {5}, {6}, {1}, {7}, {8}}
But for NetEncoder["Tokens"]
documentation states the following:
"Tokens" -> encode tokens in a string as a sequence of integer codes or one-hot vectors
What is the correct way to implement one-hot vectors? (I'm on WM 11.3)
P.S. One may do one-hot vectors encoding manually, but I need the way compatible with NetTrain
functionality:
Thread[voc -> IdentityMatrix[Length[voc]]]
{"the" -> {1, 0, 0, 0, 0, 0, 0, 0}, "quick" -> {0, 1, 0, 0, 0, 0, 0, 0}, "brown" -> {0, 0, 1, 0, 0, 0, 0, 0}, "fox" -> {0, 0, 0, 1, 0, 0, 0, 0}, "jumps" -> {0, 0, 0, 0, 1, 0, 0, 0}, "over" -> {0, 0, 0, 0, 0, 1, 0, 0}, "lazy" -> {0, 0, 0, 0, 0, 0, 1, 0}, "dog" -> {0, 0, 0, 0, 0, 0, 0, 1}}