1
$\begingroup$

I'm looking for an example of finetuning or (training from scratch) either GPT-2 or ctrl with NetTrain to generate text. Given a title, topic, url or other control strings, it will write out a reasonable paragraph.

What I've Tried:

I can find no such examples in the documentation, SSE, wolfram community, or the NN repo:

NetModel["GPT-2 Transformer Trained on WebText Data", "TrainingNet"]

enter image description here

NetModel["GPT-2 Transformer Trained on WebText Data", "ConstructionNotebook"]

enter image description here

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I'm not sure if this is exactly what you're looking for,

But the GPT-2 as implemented in mma atm is quite robust at creating things.

lm = NetModel[{"GPT-2 Transformer Trained on WebText Data", "Task" -> "LanguageModeling"}];

generateSample[languagemodel_][input_String, numTokens_: 10, temperature_: 1] := Nest[Function[StringJoin[#, languagemodel[#, {"RandomSample", "Temperature" -> temperature}]]], input, numTokens];


generateSample[lm]["https://mathematica.stackexchange.com/ is a website that", 500]

"https://mathematica.stackexchange.com/ is a website that offers a little bit of news about genetic opprobrium towards vaccines and VMAs.

Footnote:-Dr Congleton's previous focus of them all was on relying on some 40 thousand seasonally manipulated children of the USA to"

or

https://mathematica.stackexchange.com/ is a website that showcases more advanced versioning techniques.offuntimes.net lists the best alternatives that can be found for such combinations. m17r works out the deals if you remember what they profuously have. Theii offer limited expiry date on m17r b . To get some Logistics Crafts lockie if are you interested.

They quickly turn to nonsense after that...but maybe that isn't good enough.

This was an example taken from the resources cloud

However, I have found a community post that is quite in depth in training the GPT-2 it seems with the github dataset and using openais instructions , or various python examples to be more specific, and this user managed to have the current GPT-2 create infinite text adventure games.

These people recreated the GPT-2 entirely, however via python.

I would post code, but it would only be recreating entire websites.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The bottom line is that I need to use NetTrain $\endgroup$
    – M.R.
    Commented Mar 3, 2020 at 23:01
  • $\begingroup$ @M.R. As far as I know, on the resources NN, GPT-2 can not be retrained via NetTrain and a new one must be done entirely, if you're looking for something more specific, only the Tasks Feature Extraction and LanguageModeling are accepted. you'll have to follow the community post if you want to design something specific. $\endgroup$
    – DrMrstheMonarch
    Commented Mar 3, 2020 at 23:05
0
$\begingroup$

The Wolfram Neural Network Repository has a notebook called GPT2 Transformer Trained on WebText Data. In that notebook is a section where the GPT-2 model is updated (NetTrain) with training data from a movie review database.

It seems that you could use whatever kind of data you wish instead.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.