3
$\begingroup$

I have a text and vocabulary (text has two "the" entries):

list = {"the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"}
voc =  {"the", "quick", "brown", "fox", "jumps", "over", "lazy", "dog"}

And I can do integer codes with NetEncoder.

NetEncoder[{"Tokens", voc}][list]
{{1}, {2}, {3}, {4}, {5}, {6}, {1}, {7}, {8}}

But for NetEncoder["Tokens"] documentation states the following:

"Tokens" -> encode tokens in a string as a sequence of integer codes or one-hot vectors

What is the correct way to implement one-hot vectors? (I'm on WM 11.3)


P.S. One may do one-hot vectors encoding manually, but I need the way compatible with NetTrain functionality:

Thread[voc -> IdentityMatrix[Length[voc]]]

{"the" -> {1, 0, 0, 0, 0, 0, 0, 0}, "quick" -> {0, 1, 0, 0, 0, 0, 0, 0}, "brown" -> {0, 0, 1, 0, 0, 0, 0, 0}, "fox" -> {0, 0, 0, 1, 0, 0, 0, 0}, "jumps" -> {0, 0, 0, 0, 1, 0, 0, 0}, "over" -> {0, 0, 0, 0, 0, 1, 0, 0}, "lazy" -> {0, 0, 0, 0, 0, 0, 1, 0}, "dog" -> {0, 0, 0, 0, 0, 0, 0, 1}}

$\endgroup$
5
$\begingroup$

The "Class" NetEncoder outputs one-hot vectors when the "UnitVector" form is specified:

NetEncoder[{"Class", voc, "UnitVector"}][list]    

{1,0,0,0,0,0,0,0},{0,1,0,0,0,0,0,0},{0,0,1,0,0,0,0,0},

{0,0,0,1,0,0,0,0},{0,0,0,0,1,0,0,0},{0,0,0,0,0,1,0,0},

{1,0,0,0,0,0,0,0},{0,0,0,0,0,0,1,0},{0,0,0,0,0,0,0,1}}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.