5
$\begingroup$

I'd like a tutorial for text generation using GANs, specifically for names. I have a list of company names and would like the network to generate novel ones.

This is one of the only few examples of GANs in MMA:

Here's the training data:

train = CompanyData[#, "Name"] & /@ CompanyData[];
Length[train] (* 77,563 *)
RandomSample[train, 3] (* {"Microsoft", "MST Investment", "Cyberaton"} *)

Here are the starting points and resources that I've found:

$\endgroup$
5
  • 2
    $\begingroup$ Unfortunately GAN training isn't really well-supported in Wolfram Language as of 12.0. I expect they will add this functionality in future. In the meantime you can approximate a GAN by using a negative learning rate as shown in the answers to this question. $\endgroup$
    – Carl Lange
    Aug 5, 2019 at 19:22
  • 1
    $\begingroup$ But there are some examples of training the WGAN... why can't this work exactly? Just asking for a simple vanilla gan, on words of only a few characters, it can't really be impossible no? $\endgroup$
    – user5601
    Aug 5, 2019 at 19:41
  • $\begingroup$ I don't think it's impossible - just that it's not well-supported :) $\endgroup$
    – Carl Lange
    Aug 5, 2019 at 20:17
  • $\begingroup$ Can you describe why/how a GAN is needed for this? There's an example of doing character-sequence level learning using Gated Recurrent Units in the Wolfram tutorial that seems like it would do the trick.... reference.wolfram.com/language/tutorial/… I suppose you could always use this as a generator in some GAN architecture.... $\endgroup$ Oct 15, 2021 at 2:32
  • $\begingroup$ @JoshuaSchrier Because I think it's interesting and want to see the NetTrain mechanics... but I will accept any generative method... $\endgroup$
    – user5601
    Oct 15, 2021 at 19:45

1 Answer 1

9
+25
$\begingroup$

Here is a simple example that hopefully can get you started.

For simplicity, we use the names that up to 25 characters and only contains the alphabet.

allCompanyNames = 
  Select[CompanyData[#, "Name"] & /@ CompanyData[], StringQ];

vocabulary = Alphabet[]~Join~ToUpperCase[Alphabet[]]~Join~{" "};

maxLen = 25;

companyNames = With[{
    extraVoc = 
     Complement[Flatten[Characters /@ allCompanyNames], vocabulary]
    },
   Select[allCompanyNames, 
    StringLength[#] <= 25 && StringFreeQ[#, extraVoc] &]
   ];

We will use "." as the end token and pad all names to the same length

charLen = maxLen + 1;
companyNamesPadded = 
  StringPadRight[#, charLen, "."] & /@ companyNames;
vocLen = Length[vocabulary] + 1;

This are what the padded names look like

RandomSample[companyNamesPadded, 3]
(*{"Azot PJSC.................", "Vinnytsyamyaso............", "Macofil..................."}*)

Now define the generator and the discriminator. To make it simple, we only use linear layers.

discriminator = 
 NetChain[{LinearLayer[256, "Input" -> {charLen, vocLen}], 
   ElementwiseLayer["SELU"], DropoutLayer[0.4], LinearLayer[128], 
   ElementwiseLayer["SELU"], DropoutLayer[0.4], LinearLayer[{}], 
   LogisticSigmoid}, "Input" -> {charLen, vocLen}]

latenLen = 100;
generator = NetFlatten@NetChain[{
        LinearLayer[{charLen, 512}], ElementwiseLayer["SELU"], 
    DropoutLayer[0.3], LinearLayer[{charLen, 256}], 
    ElementwiseLayer["SELU"], DropoutLayer[0.3], 
    LinearLayer[{charLen, 128}], ElementwiseLayer["SELU"], 
    LinearLayer[{charLen, vocLen}], ElementwiseLayer["SELU"], 
    SoftmaxLayer[]
    }, "Input" -> latenLen]

Define a data generator that yields random seeds and sampled names.

getRandomLatent[batchSize_] := 
 Map[NumericArray[#, "Real32"] &, 
  RandomVariate[NormalDistribution[], {batchSize, latenLen}]]
datagen = 
  Function[<|
    "Sample" -> 
     UnitVectorLayer[vocLen][
      NetEncoder[{"Characters", {vocabulary, "."}}][
       RandomSample[companyNamesPadded, #BatchSize]]], 
    "Latent" -> getRandomLatent[#BatchSize]|>];

Train the network

trained = NetTrain[NetGANOperator[{generator, discriminator}],
  {datagen, "RoundLength" -> Length[companyNamesPadded]},
  TrainingUpdateSchedule -> {"Discriminator", "Generator"},
  MaxTrainingRounds -> 60, TargetDevice -> "GPU", BatchSize -> 512
  ]

Extract the generator

trainedgen = NetExtract[trained, "Generator"]
dec = NetDecoder[{"Characters", {vocabulary, "."}}]

Generate names by sampling the latent space

Table[First@
  StringSplit[
   dec@trainedgen[RandomVariate[NormalDistribution[], latenLen]], 
   "."], {10}]
(*{"Lokeirrle", "Lracian", "Lofcia", "Meritian", "Siriar", "Palteo", "Garirlso ", "Loepia", "Lraedm a", "Demiiiaa"}*)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.