I have a text data, which is a list of different sentences. I used below for example:-

dataAll = ResourceData["State of the Union Addresses"];
data = TextSentences[dataAll[1]["Text"]];

Then I defined a FeatureExtraction function fe, and then I input a sentence as a simple test and we can see from the output that the dimension of output is 23, which means we would extract 23 features from the sentence:

fe = FeatureExtraction[data, {"NumericVector"}];
Length@fe["This is a test"]

To do this, we need to know what's going on inside FeatureExtraction. So we use this to have a look:-

processList = Options[fe][[1]]["Processor"][[2]]["Processors"]

We can see that fe is, in fact, applying TFIDF to the word bag, and obtain a large vector with dimensions say several thousand. And they fe would apply Dimension reduction twice and finally obtained a vector with smaller dimension of 23.

I can see the details of the TFIDF and dimension reduction from the below codes:-

parametersTFIDF = normSqToTFIDF[[2]]["InverseDocumentFrequency"];
dimWordBag = Length@parametersTFIDF
dimRedNumV = processList[[5]];

What I want to do is, to use a sentence as input, and then obtain a feature vector of different dimensions, say 19 instead of 23. How can I make a surgery to fe to achieve this?

If it is difficult to amend fe, it's okay to me to proceed by using another self-defined functions to achieve the result.

Many thanks!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.