Here is another option:


    hexToRGB =
      RGBColor[ FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}] ] &;
    
    hexToRGB@"#FF8c00" // ColorSetter

![Mathematica graphics](https://i.sstatic.net/ceXCx.png)

And another:

    hexToRGB =
      RGBColor[ FromDigits[#, 16]/255 & /@ StringCases[#, Except["#"] ~~ _] ] &;


----------

## `#RGB` form

Responding to cormullion's comment:

    hexToRGB[color_String | {colors__String}] :=
      RGBColor[FromDigits[#, 16]/255 & /@ #] & @@@
        StringCases[{color, colors},
          {"#" ~~ r_ ~~ g_ ~~ b_ ~~ EndOfString :> {r ~~ r, g ~~ g, b ~~ b},
           "#" ~~ r : (_ ~~ _) ~~ g : (_ ~~ _) ~~ b : (_ ~~ _) :> {r, g, b}}]

    ColorSetter /@ hexToRGB @ 
      {"#000", "#FF0000", "#0F0", "#0000FF", "#FF0", "#00FFFF", "#F0F", "#C0C0C0", "#FFF"}

![Mathematica graphics](https://i.sstatic.net/sXsQQ.png)

Operating on the entire list of color strings should be faster than one at a time.