I'm trying to understand why I can decode encoded word tokens?

s = "Why isn't this decoding?";
e = NetEncoder[{"Tokens", "English", "IgnoreCase" -> True}];
d = NetDecoder@e;
encoded = e@s;

enter image description here

  • $\begingroup$ Instead of using 40236, you may need to change the number. There should be an error message if it's wrong indicating the right number. The underlying list of words may change. $\endgroup$
    – Searke
    Jun 14, 2019 at 16:33

1 Answer 1


Ah, I missed this in the documentation:

The input to the decoder input is either a vector of probabilities or a sequence of probability vectors. Each probability vector sums to 1. The length of each probability vector is the size of the vocabulary.

Should be:

d[UnitVector[40236, #] & /@ encoded]

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.