How to view ClassifierFunction's preprocessed data?

Question

Classify apparently preprocesses your input data, for example (same example from Classify reference page, but using NeuralNetwork as the method):

Othello = Import["http://www.gutenberg.org/cache/epub/2267/pg2267.txt"];
Hamlet = Import["http://www.gutenberg.org/cache/epub/2265/pg2265.txt"];
Macbeth = Import["http://www.gutenberg.org/cache/epub/2264/pg2264.txt"];
TheImportanceOfBeingEarnest = Import["http://www.gutenberg.org/cache/epub/844/pg844.txt"];
ThePictureofDorianGray = Import["http://www.gutenberg.org/cache/epub/174/pg174.txt"];
AnIdealHusband = Import["http://www.gutenberg.org/files/885/885-0.txt"];
LesMiserables = Import["http://www.gutenberg.org/cache/epub/135/pg135.txt"];
NotreDamedeParis = Import["http://www.gutenberg.org/cache/epub/2610/pg2610.txt"];
TheManWhoLaughs = Import["http://www.gutenberg.org/cache/epub/12587/pg12587.txt"];

author = Classify[
           <|"William Shakespeare" -> {Othello, Hamlet},
             "Oscar Wilde" -> {TheImportanceOfBeingEarnest, ThePictureofDorianGray},
             "Victor Hugo" -> {LesMiserables, NotreDamedeParis}|>,
           Method -> {"NeuralNetwork"}];

Now, by executing Options[author], we get more information about the trained Neural Network:

{<|"Common"-><|"Pattern"->_String,"Preprocessors"-><|"Input"->"ToLowerCase" -> "WordTokenize" -> "TokensToTFIDFVector" -> "ReduceDimension","Output"->"IntegerEncodeList"|>,"Decision"-><|"Prior"-><|"Oscar Wilde"->0.3333333333333333`,"Victor Hugo"->0.3333333333333333`,"William Shakespeare"->0.3333333333333333`|>...

I'm wondering if there's any way we can see what the preprocessed data is (for example: see what the TFIDF vector or IntegerEncodeList look like).

Answer 1

The PrintDefinitions function may be helpful in understanding the internal working of the processors.

It seems that the internal structure of data is in the form of MLDataset, and the processors can be applied directly to the MLDataset.

Here is an example:

We first extract the options, which contains the detail of the model

opts = First@Options[author];

The preprocessor and processor of the input can be obtained by

inputPrepro = opts["Input"]["Preprocessor"]
inputPro = opts["Input"]["Processor"]

Now we can apply these processors to the inputs, and get an MLDataset object

processedInput = inputPro@inputPrepro@Keys[data]
(* MachineLearning`MLDataset[<|"f1" -> <|"Type" -> "Text", 
    "Weight" -> 1, 
    "Values" -> {"william shakespeare", "oscar wilde", "victor hugo"},
     "ID" -> 5544833639454981494|>|>, <|"RawExample" -> False, 
  "ExampleNumber" -> 3, "ExampleWeights" -> 1|>] *)

Then we can apply the processor in the model to this MLDataset object

modelPro = First[opts["Models"]]["Processor"];
modelPro@processedInput
(* {{2.71499}, {2.7148}, {2.71468}} *)

If you want to look into the individual processor in the modelPro, you can apply the individual processors sequentially

{pro1, pro2, pro3, pro4} = modelPro[[2]]["Processors"];

pro4@pro3@pro2@pro1@processedInput
(* {{2.71499}, {2.7148}, {2.71468}} *)