How to paraphrase text?

Question

I'm looking to train or use pre-trained models in order to re-phrase text. Are there any natural language models in the Wolfram Neural Repository (or elsewhere) that can be used to paraphrase text?

Most text rephrasing models are not conditional, e.g. what they typically do is translate your text to another language and then back, and this will vary some of the words and phrases. But it’s deterministic so there’s no random-seed to provide or any good way to control the amount of variation.

I'm looking for any links/leads or examples for paraphrasing in Mathematica.

References:

• A paraphrase detection competition to identify question pairs with the same intent/meaning: https://www.kaggle.com/c/quora-question-pairs
• Code that implements various paraphrase identification models in tensorflow: nelson-liu/paraphrase-id-tensorflow (https://github.com/nelson-liu/paraphrase-id-tensorflow)
• Paraphrase Identification Models blogenter link description here.

Answer 1

Update - some thoughts on Neural Nets

You can think of using bidirectional, language model and text generation networks somehow altering their architecture. For instance, BERT's underlying idea is to to fill a blank in context like:

"It is raining ____ and dogs today"

so it would yield "cats" or maybe "cows" if it is a bit amok :-) So you can randomly remove words form your sentence and run it through NN to fill in the gap with a similar term. You can also consider some NN architecture able to train on dataset of the type string-> string. You would use deterministic language translation to build a training dataset of the type

{.., "senttence1"->"paraphrase11", "senttence1"->"paraphrase12"...,"senttence2"->"paraphrase21", "senttence2"->"paraphrase22",...}

And then train a NN on it which will not be deterministic. But this requires advanced playing with NN architecture. NNs to consider:

• BERT: https://resources.wolframcloud.com/NeuralNetRepository/resources/BERT-Trained-on-BookCorpus-and-English-Wikipedia-Data

• ELLMo: https://resources.wolframcloud.com/NeuralNetRepository/resources/ELMo-Contextual-Word-Representations-Trained-on-1B-Word-Benchmark

• GTP2: https://resources.wolframcloud.com/NeuralNetRepository/resources/GPT2-Transformer-Trained-on-WebText-Data

Non-neural net

Here are simple toy-model idea, non-neural. Real application might need more heuristics and tuning, but might turn out better or simpler than a neural net. Synonyms can go pretty wild and jump word classes say from noun to verb etc. A simple trick is to find in original sentence a word class and then narrow down your synonyms accordingly. WordData can do this:

WordData[{"crazy", "Noun"}, "Synonyms", "List"]
Out[]= {"looney", "loony", "nutcase", "weirdo"}

compare:

WordData[{"crazy", "Adjective"}, "Synonyms", "List"]
Out[]= {"brainsick","demented","disturbed","dotty","gaga","half-baked","mad","screwball","sick","softheaded","unbalanced","unhinged","wild"}

You can define a randomized function:

paraphrase[sentence_,type_]:=
StringReplace[sentence,
#->RandomChoice[{#}~Join~WordData[{#,type},"Synonyms","List"]]&/@
TextCases[sentence,type]]

so given an original sentence:

sent="previously, tea had been used primarily for Buddhist monks to stay awake during meditation.";

you can sequentially replace various word cllasses"

Table[Fold[paraphrase,sent,{"Adjective","Adverb","Noun"}],5]//Column

This is very far from ideal. Note that you needd to add original word in case the synonyms list comes up empty. This is also based on WordNet which is a bit weird synonym-wise. You might use other means like

EntityValue[Entity["Word", "mad"], "SynonymsList"]

Out[]= {"huffy","sore","brainsick","crazy","demented","disturbed","sick","unbalanced","delirious","excited","frantic","unrestrained","harebrained","insane"}

or various available external APIs. But these might be harder to narrow down for a word class. Also note word capitalization is tricky. For WordData you need lower case mostly except proper nouns, so beginning of sentences is tricky.