This could provide a good starting point:

First step is to provide multiple examples of "red", "blue", etc for the training set. I found a site that had some convenient images for this task. For example a good of "reds":

So I import the images:

images = Import[
   "http://ingridsundberg.com/2014/02/04/the-color-thesaurus/", 
   "Images"][[2 ;; 13]];

assign some labels:

names = {"white", "tan", "yellow", "orange", "red", "pink", "purple", 
         "blue", "green", "brown", "gray", "black"}

pull out the "dominant" colors:

colors=DominantColors[#, 20, ColorCoverage -> 1/30, MinColorDistance -> 0] & /@ images;

and then train (but first I remove the RGBColor head as I had some difficulties passing RGBColor[r,g,b] directly to Classify)

training = Apply[List,colors,2];
cf = Classify[<|Thread[names -> training]|>];

from there I get some decent results:

cf[List @@ Brown]
(* "brown" *)

cf[List @@ Orange]
(* "orange" *)

cf[{0.1, 0.25, 0}]
(* "green" *)

Manipulate[cf[List @@ color], {color, Red}]

This could provide a good starting point:

First step is to provide multiple examples of "red", "blue", etc for the training set. I found a site that had some convenient images for this task. For example a good sample of "reds":

So I import the images:

images = Import[
   "http://ingridsundberg.com/2014/02/04/the-color-thesaurus/", 
   "Images"][[2 ;; 13]];

assign some labels:

names = {"white", "tan", "yellow", "orange", "red", "pink", "purple", 
         "blue", "green", "brown", "gray", "black"}

pull out the "dominant" colors:

colors=DominantColors[#, 20, ColorCoverage -> 1/30, MinColorDistance -> 0] & /@ images;

and then train (but first I remove the RGBColor head as I had some difficulties passing RGBColor[r,g,b] directly to Classify)

training = Apply[List,colors,2];
cf = Classify[<|Thread[names -> training]|>];

from there I get some decent results:

cf[List @@ Brown]
(* "brown" *)

cf[List @@ Orange]
(* "orange" *)

cf[{0.1, 0.25, 0}]
(* "green" *)

Manipulate[cf[List @@ color], {color, Red}]