Also
rgb[x_] := RGBColor[FromDigits[#, 2] / 255 & /@
Partition[IntegerDigits[FromDigits[StringDrop[x, 1], 16], 2, 24], 8]]
rgb@"#FF5500"
RGBColor[{255, 85, 0}]
Edit
Golfing a one liner :)
rgb[x_] := RGBColor[FromDigits[#, 16]/255 & /@ StringJoin /@ Partition[Rest@Characters@x, 2]]