4
$\begingroup$

I'm attempting to do simple category recommendation on textual data. I have around 20k training samples of the form desc_String -> {tag1_String, tag2_String, ...}.

When given a natural language input (in my case a product description string), the trained NetChain should produce a probability distribution over the category tags.

Here's a minimal example of the issue:

data = {
   "Axel Skin Tight Leggings for the Outdoors" -> {"summer-clothes", "skinny"}, 
   "Velvet Mini Skirt" -> {"summer-clothes", "bottoms"}, 
   "Stretch Cotton Scarf" -> {"winter-clothes", "accessories"},
   "California Striped Colorblock Bomber Jacket " -> {"sale-outerwear", "tops", "winter-clothes"}};

allLabels = Union @ Flatten[data[[All, 2]]]
dim = Length @ allLabels

net = NetChain[{
         EmbeddingLayer[60], DropoutLayer[0.3], 
         LongShortTermMemoryLayer[40], SequenceLastLayer[], 
         LinearLayer[dim], SoftmaxLayer[]
       },
      "Input" -> NetEncoder[{"Tokens"}], 
      "Output" -> NetDecoder[{"Class", allLabels}]
  ]

trained = NetTrain[net, data]

enter image description here

The NetTrain seems to complain that each sample's target is a list and not a string, but this is exactly what multi-label training is.

I'm sure NetTrain and En/decoders must support multi-labels, so what am I doing wrong here?

Updates:

It was suggested in the comment to convert my training data's labels into one-hot vectors. This clearly works for "Inputs" (e.g. UnitVectorLayer) however, I've never seen this work for target "Outputs".

$\endgroup$
  • $\begingroup$ I'm not sure it's supported. But you can make your data single labeled and just extract top probability outputs from your network. $\endgroup$ – swish Apr 17 '18 at 20:00
  • $\begingroup$ But then the set-wise structural information will be lost, and the dimensionally would explode $\endgroup$ – M.R. Apr 17 '18 at 20:01
  • $\begingroup$ Would it though? Isn't training on several identical examples with different labels the same as training on one with multiple labels? Another solution is to provide labels as vector in the form of {0,1,0,0,1} with ones being in place of current example labels. $\endgroup$ – swish Apr 17 '18 at 20:08
  • $\begingroup$ No it’s not the same, there is more information content in the list $\endgroup$ – M.R. Apr 17 '18 at 20:10
  • $\begingroup$ Yes the 1-hot vectorized form is equivalent $\endgroup$ – M.R. Apr 17 '18 at 20:11
4
$\begingroup$

Make your data multi-hot:

dataMultiHot = 
 MapAt[Total@*NetEncoder[{"Class", allLabels, "UnitVector"}], 
  data, {All, 2}]

Use LogisticSigmoid as last activation:

net = NetInitialize@
  NetChain[{EmbeddingLayer[60], DropoutLayer[0.3], 
    LongShortTermMemoryLayer[40], SequenceLastLayer[], 
    LinearLayer[dim], LogisticSigmoid}, 
   "Input" -> NetEncoder[{"Tokens"}], 
   "Output" -> NetDecoder[{"Class", allLabels}]]
net = NetTrain[net, dataMultiHot]

Output confidence for each class:

net["Winter scarf", "Probabilities"]

(*<|"accessories" -> 0.708859, "bottoms" -> 0.200409, 
 "sale-outerwear" -> 0.14959, "skinny" -> 0.168188, 
 "summer-clothes" -> 0.220286, "tops" -> 0.153834, 
 "winter-clothes" -> 0.784767|>*)

UPDATE

In order for network to output only 0's and 1's, you can either stick additional ElementwiseLayer[Round] at the end and drop NetDecoder altogether

net = NetInitialize@
  NetChain[{EmbeddingLayer[60], DropoutLayer[0.3], 
    LongShortTermMemoryLayer[40], SequenceLastLayer[], 
    LinearLayer[dim], LogisticSigmoid, ElementwiseLayer[Round]}, 
   "Input" -> NetEncoder[{"Tokens"}]]

or try to learn thresholds separately:

net = NetInitialize@NetGraph[{
    EmbeddingLayer[60], DropoutLayer[0.3], 
    LongShortTermMemoryLayer[40], SequenceLastLayer[], 
    LinearLayer[dim], LogisticSigmoid, LinearLayer[dim], 
    LogisticSigmoid, 
    ThreadingLayer[(Sign[#1 - #2] + 1)/2&]}, {1 -> 
     2 -> 3 -> 4 -> 5 -> 6, 4 -> 7 -> 8, {6, 8} -> 9}, 
   "Input" -> NetEncoder[{"Tokens"}]]

There is a limitations on what functions you can use inside ElementwiseLayer and ThreadingLayer, Sign works but not UnitStep or Piecewise or If etc., that's why it looks weird.

Then you extract labels from vector:

Extract[allLabels, Position[net["Winter scarf"], 1.0]]
$\endgroup$
  • $\begingroup$ this looked right but then I realized the net only outputs a single string, but it's multi-output so I'd like it to output multiple strings. Is there any NetDecoder that can do that, that is, learn to threshold the LogisticSigmoid vector and return the labels for those indexes? $\endgroup$ – M.R. Apr 18 '18 at 23:01
0
$\begingroup$

Not sure this is what you need but:

assign[a_, listb_] := Map[a -> # &, listb]

You can make your data single-labeled thus:

Flatten[Map[assign[Keys[#], Values[#]] &, data]]

{"Axel Skin Tight Leggings for the Outdoors" -> "summer-clothes", "Axel Skin Tight Leggings for the Outdoors" -> "skinny", "Velvet Mini Skirt" -> "summer-clothes", "Velvet Mini Skirt" -> "bottoms", "Stretch Cotton Scarf" -> "winter-clothes", "Stretch Cotton Scarf" -> "accessories", "California Striped Colorblock Bomber Jacket " -> "sale-outerwear", "California Striped Colorblock Bomber Jacket " -> "tops", "California Striped Colorblock Bomber Jacket " -> "winter-clothes"}

and run the training as you did:

allLabels = Union@Flatten[data[[All, 2]]] dim = Length@allLabels

net = NetChain[{EmbeddingLayer[60], DropoutLayer[0.3], LongShortTermMemoryLayer[40], SequenceLastLayer[], LinearLayer[dim], SoftmaxLayer[]}, "Input" -> NetEncoder[{"Tokens"}], "Output" -> NetDecoder[{"Class", allLabels}]]

trained = NetTrain[net, Flatten[Map[assign[Keys[#], Values[#]] &, data]]]

For the resulting probabilities you can do:

Thread[ List[allLabels, trained["Axel Skin Tight Leggings for the Outdoors", None]]]

{{"accessories", 7.54239*10^-6}, {"bottoms", 0.0016615}, {"sale-outerwear", 0.000471794}, {"skinny", 0.492985}, {"summer-clothes", 0.503679}, {"tops", 0.00100407}, {"winter-clothes", 0.000190549}}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.