0
$\begingroup$

I am looking to weight the words (TF-IDF) of a random text by his occurence and showing that on a matrix. I saw there is a project on it but would like to know if it possible to change the visualization ? https://demonstrations.wolfram.com/TermWeightingWithTFIDF/

Example of the final output enter image description here

$\endgroup$

closed as unclear what you're asking by Daniel Lichtblau, m_goldberg, Henrik Schumacher, eyorble, Pinti Apr 12 at 11:48

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ It’s very likely that you could. But what would you like to change it to? What is your code so far? $\endgroup$ – MarcoB Apr 7 at 12:52
  • $\begingroup$ So far I only clean the data I have by removing stopwords and punctuation... The next step I would have is to : 1. separate my dataset (one big text) into subsets representing : {sentence 1}, {sentence 2...}.... maybe by attributing for each sentence an ID ? 2. taking a unique list of all the words in the text 3. Taking each sentence and count for each word if the word appear in the sentence 4. Put the result under a table like above $\endgroup$ – Tom Peterson Apr 7 at 13:26
  • $\begingroup$ Welcome to Mathematica.SE! I suggest the following: 1) As you receive help, try to give it too, by answering questions in your area of expertise. 2) Take the tour! 3) When you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. Also, please remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign! $\endgroup$ – Michael E2 Apr 8 at 15:29
  • $\begingroup$ Possible duplicate of Words weighting with TF-IDF (updated with cleaned code) $\endgroup$ – Pinti Apr 12 at 11:48
0
$\begingroup$

From the comments of the question:

So far I only clean the data I have by removing stopwords and punctuation... The next step I would have is to : 1. separate my dataset (one big text) into subsets representing : {sentence 1}, {sentence 2...}.... maybe by attributing for each sentence an ID ? 2. taking a unique list of all the words in the text 3. Taking each sentence and count for each word if the word appear in the sentence 4. Put the result under a table like above

This process is more or less followed in the blog post: "The Great conversation in USA presidential speeches".

  1. The crucial step is making the document-word contingency matrix. Generally speaking, for this you can use the function CrossTabulate from the package CrossTabulate.m. For more details see this blog post: "Contingency tables creation examples".

  2. The TF-IDF and related measures can be computed with the package DocumentTermMatrixConstruction.m.

(These packages are used in the blog post "The Great conversation in USA presidential speeches".)

$\endgroup$
1
$\begingroup$

"I am looking to weight the words (TF-IDF) of a random text by his occurrence and showing that on a matrix."

Weighting words by frequency of occurrence is the same as normalizing the columns of the matrix. To do this, let data be you term/document matrix, and sum the columns by using Total. The final line divides each column by the appropriate sum.

data = RandomInteger[{0, 1}, {5, 15}];
norm = Total[data] /. {0 -> 1};
tab = Transpose[Table[Transpose[data][[i]]/norm[[i]], {i, Length[norm]}]];
TableForm[tab]

enter image description here

You'll have to decide what you want it to look like.

$\endgroup$
  • $\begingroup$ I come up with this code by now : TFIDF = FeatureExtraction[ Join[First /@ Keys@ruleData[[All]], Last /@ Keys@ruleData[[All]]], "TFIDF"] but the output is telling me that there is nonatomic expression my variable rule above is an association between label and sentences ( 1--> sentence 1, 2--> sentence2, 3--> sentence3...) As I wanted to apply the tf-idf for each sentence and for the total of sentences $\endgroup$ – Tom Peterson Apr 8 at 13:32
0
$\begingroup$

I come up with this code by now :

TFIDF = FeatureExtraction[ Join[First /@ Keys@ruleData[[All]], Last /@ Keys@ruleData[[All]]], "TFIDF"]

but the output is telling me that there is nonatomic expression

my variable rule above is an association between label and sentences ( 1--> sentence 1, 2--> sentence2, 3--> sentence3...) As I wanted to apply the tf-idf for each sentence and for the total of sentences

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.