Also

    rgb[x_] := RGBColor[FromDigits[#, 2] / 255 & /@ 
                Partition[IntegerDigits[FromDigits[StringDrop[x, 1], 16], 2, 24], 8]]

    rgb@"#FF5500"

>RGBColor[{255, 85, 0}]


**Edit** 

Golfing a one liner :)

    rgb[x_] := RGBColor[FromDigits[#, 16]/255 & /@ StringJoin /@ Partition[Rest@Characters@x, 2]]