Weighted Data within Classify and Predict (or NetTrain)

Question

I posed this on Wolfram Community about a week ago and no one there seems to have a good solution, so I will try it here. (It may well be that this is an issue for Mathematica developers and that little can be done at present, but ...)

It does not appear easy within the current Classify, Predict or NetTrain functions to accommodate weighted data. (Note, I am not speaking about class priors but about what are sometimes called case weights.) It I am wrong about this, can anyone tell me how to do so. If anyone has a work around, could you please make a suggestion.

Motivation: I have several projects that would benefit from weighted data, the latest being to explain this really interesting paper using Mathematica: https://arxiv.org/pdf/1602.04938.pdf . The idea would basically be to emulate one machine learning method -- the complex one such as a neural net -- with a less opaque one (such as a decision tree) but with the emulator only being responsible for producing similar answers within a neighborhood of a user-specified point. Closer points within that neighborhood would be weighted more heavily than farther-away points within the neighborhood. This way one could use something close to human language to construct an approximate but comprehensible explanation, for a particular individual, what a more complex and more opaque classifier was doing.

I've thought about random sampling of points within the neighborhood with the weights determined by distance, but I would prefer a more deterministic algorithm.

Answer 1

Your question is very interesting. And I hope that you will find something useful in my answer.

My idea is to do multi-task learning: classification and learning embedding.

SeedRandom[1234];
X = RandomReal[{-1, 1}, {1000, 10}];
Y = RandomInteger[{0, 1}, 1000];

net = NetGraph[
  {
   50,
   50,
   CatenateLayer[],
   ReshapeLayer[{2, 50}],
   NetPairEmbeddingOperator[NetChain[{10, Ramp, 2(*embedding dimension*)}]],
   {2, SoftmaxLayer[]},
   {2, SoftmaxLayer[]},
   ContrastiveLossLayer[(*margin*)]
   },
  {
   NetPort["Input1"] -> 1 -> 6,
   NetPort["Input2"] -> 2 -> 7,
   {1, 2} -> 3 -> 4 -> 5 -> 8
   },
  "Input1" -> 10,
  "Input2" -> 10,
  "Output1" -> NetDecoder[{"Class", {0, 1}}],
  "Output2" -> NetDecoder[{"Class", {0, 1}}]
  ]

sampler[n_] := Block[
  {r},
  r = Table[RandomSample[Range[1000], 2], n];
  <|
   "Input1" -> X[[r[[;; , 1]]]],
   "Input2" -> X[[r[[;; , 2]]]],
   "Output1" -> Y[[r[[;; , 1]]]],
   "Output2" -> Y[[r[[;; , 2]]]],
   "Target" -> RandomChoice[{True, False}, n]
   |>
  ]

Replace RandomChoice for Target with your own function which will measure closeness of the points.

This sampler doesn't work inside of NetTrain and I don't know why.

sampler[#BatchSize] &@<|"BatchSize" -> 2|>
netT = NetTrain[net, sampler[#BatchSize] &, BatchSize -> 2, MaxTrainingRounds -> 50]

NetTrain::invgenout: Output of generator function sampler[Slot[<<1>>]]& was incorrect: specification for slot Output1 is missing.

So temporary we will use sampler as the dataset generator and will set n as big as we want.

netT = NetTrain[net, sampler[100000], MaxTrainingRounds -> 10]

Now we have the trained network. We have two classifiers in our network. We can group them into the one ensemble and improve accuracy.

classifier = NetGraph[
  {
   Take[netT, {1, 6}],
   Take[netT, {2, 7}],
   ThreadingLayer[(#1 + #2)/2 &]
   },
  {
   {1, 2} -> 3
   },
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

Now the most important part. We extract our embedding network and create NearestFunction with our embeddings.

emb1 = NetChain[
  {
   Take[netT, {1, 1}],
   NetExtract[netT, 5][["Net"]]
   }
  ]

nf1 = Nearest[emb1@X -> Range[n]]

emb2 = NetChain[
  {
   Take[netT, {2, 2}],
   NetExtract[netT, 5][["Net"]]
   }
  ];
nf2 = Nearest[emb2@X -> Range[n]];

For example, we have the new data point.

SeedRandom[0];
data = RandomReal[{-1, 1}, 10];

Classification:

classifier[<|"Input1" -> data, "Input2" -> data|>, "Probabilities"]

<|0 -> 0.603743, 1 -> 0.396257|>

5 nearest examples in our training data:

{nf1[emb1@data, 5], nf2[emb2@data, 5]}

{{397, 120, 216, 377, 908}, {82, 418, 242, 896, 336}}

And now we can correct predicted probabilities or to do something else.

You can read more about multi-task learning, hard and soft parameter sharing here:

An Overview of Multi-Task Learning in Deep Neural Networks