0
$\begingroup$

I have a text data, which is a list of different sentences. I used below for example:-

dataAll = ResourceData["State of the Union Addresses"];
data = TextSentences[dataAll[1]["Text"]];

Then I defined a FeatureExtraction function fe, and then I input a sentence as a simple test and we can see from the output that the dimension of output is 23, which means we would extract 23 features from the sentence:

fe = FeatureExtraction[data, {"NumericVector"}];
Length@fe["This is a test"]

To do this, we need to know what's going on inside FeatureExtraction. So we use this to have a look:-

processList = Options[fe][[1]]["Processor"][[2]]["Processors"]

We can see that fe is, in fact, applying TFIDF to the word bag, and obtain a large vector with dimensions say several thousand. And they fe would apply Dimension reduction twice and finally obtained a vector with smaller dimension of 23.

I can see the details of the TFIDF and dimension reduction from the below codes:-

parametersTFIDF = normSqToTFIDF[[2]]["InverseDocumentFrequency"];
dimWordBag = Length@parametersTFIDF
dimRedNumV = processList[[5]];
Options[dimRedNumV]

What I want to do is, to use a sentence as input, and then obtain a feature vector of different dimensions, say 19 instead of 23. How can I make a surgery to fe to achieve this?

If it is difficult to amend fe, it's okay to me to proceed by using another self-defined functions to achieve the result.

Many thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.