The multi-armed bandit problem (MAB) is a classic in Reinforcement learning. Now that Mathematica version 12 finally has support for Reinforcement learning:
- Here's a nice "mnist example" for training a policy network for the cartpole problem.
In the style of this question, I'm asking is it possible to implement a multi-arm bandit using the new neural network framework in Version 12?
An Example Dataset to Play with
Here's a simple synthetic dataset of recommendations consisting of 500k rows each with three items: which algorithm the bandit chose for a given user (algo A or B), a vector of that user's (possibly missing) representative features, and lastly, whether or not the user interacted with whatever content the bandit's chosen algorithm presented:
d = CloudGet @ "https://www.wolframcloud.com/obj/21a37a6b-ab48-4014-88e3-1150b2b4429f";
d // Transpose
Notice that there are many more examples (this is typically the case) where no interaction occurred:
So that means training the bandit as a Classifier would require using ClassPriors and possibly weighting or augmenting the positive training data samples.
A bit of background on bandits:
MABs are what Netflix uses to choose the tiles that they show each of their users, check out their insightful blog post. Here's the basic idea:
The name comes from imagining a gambler at a row of slot machines (sometimes known as "one-armed bandits", each arm having its own rigged probability distribution), who has to decide which machines to play, how many times to play each machine and in which order to play them, and whether to continue with the current machine or try a different machine.
The contextual or multi-armed bandit problem is slightly different: in each iteration, an agent has to choose between arms. Before making the choice, the agent sees a d-dimensional feature vector (i.e. the context), and after the choice is made the environment gives a reward. Over many iterations, the agent learns a predictor function that takes a context and returns the next best arm to play. A simple and effective way to implement a MAB is with Thompson Sampling which uses Bayesian logic to maximize expected rewards.
Now that RL in MMA is possible, I'm hoping someone can provide examples of implementing and training contextual bandits in MMA. There are many pre-trained networks in the repository perhaps, perhaps some are applicable in reinforcement learning?
Notes:
- A Bandit algorithm is a stochastic scheduler for online optimization and there are many variants: Dueling, Adversarial, Collaborative bandits, etc. but I'm only interested in contextual ones
- Some known MAB algorithms: https://www.cs.mcgill.ca/~vkules/bandits.pdf
- A few nice tutorials on MABs:
