Here is another option:
hexToRGB =
RGBColor[ FromDigits[#, 16]/255 & /@ StringTake[#, {{2, 3}, {4, 5}, {6, 7}}] ] &;
hexToRGB@"#FF8c00" // ColorSetter
And another:
hexToRGB =
RGBColor[ FromDigits[#, 16]/255 & /@ StringCases[#, Except["#"] ~~ _] ] &;