We make plot image size a control variable and use the (undocumented) Method
option "ControlAreaDisplayFunction"
to adjust the horizontal size of the display area. The display area content is wrapped in Pane
with the option Alignment -> {Center, Center}
to make display area content stay centered.
Manipulate[Plot[x Sin[x], {x, 0, 2 Pi}, PlotStyle -> color,
ImageSize -> Dynamic[imagesize]],
{{color, Red}, None},
Button["A", color = color /. {Red->Green, Green->Red},
ImageSize -> {Scaled[.1], Automatic}],
{{imagesize, {300, 250}}, None},
Method -> {"ControlAreaDisplayFunction" ->
(Dynamic @ Pane[#, Alignment -> {Center, Center},
ImageSize -> {First @ imagesize, Automatic}] &)}]

Manipulate[Plot[x Sin[a x + b], {x, 0, 2 Pi}, PlotStyle -> color,
ImageSize -> Dynamic[imagesize]],
Column[{Column[{Control@{a, -1, 1},
Control@{b, {1, 2, 3} Pi/4},
Control @ {{color, Red}, ColorSlider}}],
Button["A", color = Darker @ color, ImageSize -> {30, Automatic}]},
Alignment -> Center],
{{imagesize, {300, 250}}, None},
Method -> {"ControlAreaDisplayFunction" ->
(Dynamic @ Pane[#, Alignment -> {Center, Center},
ImageSize -> {First @ imagesize, Automatic} &)}]

Row[{Spacer[ImageDimensions[g][[1]]/2], Button["A"]}]
whereg = Plot[Sin[x], {x, 1, 4}]
is not really a solution, since it´, indeed, places that button centered, but does not react dynamically. $\endgroup$