Return to Answer

Here is another option:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}] ] &;

hexToRGB@"#FF8c00" // ColorSetter

And another:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringCases[#, Except["#"] ~~ _] ] &;

Here is another option:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}] ] &;

hexToRGB@"#FF8c00" // ColorSetter

And another:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringCases[#, Except["#"] ~~ _] ] &;

#RGB form

Responding to cormullion's comment:

hexToRGB[color_String | {colors__String}] :=
  RGBColor[FromDigits[#, 16]/255 & /@ #] & @@@
    StringCases[{color, colors},
      {"#" ~~ r_ ~~ g_ ~~ b_ ~~ EndOfString :> {r ~~ r, g ~~ g, b ~~ b},
       "#" ~~ r : (_ ~~ _) ~~ g : (_ ~~ _) ~~ b : (_ ~~ _) :> {r, g, b}}]

ColorSetter /@ hexToRGB @ 
  {"#000", "#FF0000", "#0F0", "#0000FF", "#FF0", "#00FFFF", "#F0F", "#C0C0C0", "#FFF"}

Operating on the entire list of color strings should be faster than one at a time.

Here is another option:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}] ] &;

hexToRGB@"#FF8c00" // ColorSetter

Here is another option:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}] ] &;

hexToRGB@"#FF8c00" // ColorSetter

And another:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringCases[#, Except["#"] ~~ _] ] &;

Here is another option:

hexToRGB =
  RGBColor @@ (FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}]) &;

hexToRGB@"#FF8c00" // ColorSetter

Here is another option:

hexToRGB =
  RGBColor[ FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}] ] &;

hexToRGB@"#FF8c00" // ColorSetter