Using graphics primitives as buttons

Question

I am trying to use graphics primitives as buttons, but can't do it unless I rasterise first:

button[letter_] := 
Button[
    Style[letter, Bold, 30],
    Appearance -> {"Default" -> #, "Hover" -> #2, "Pressed" -> #3}] & @@ 
    Table[
        Rasterize[
            Graphics[{
                EdgeForm[None], c, Polygon[CirclePoints[6]]
            }],
        ImageSize->100],
        {c, {GrayLevel[#], GrayLevel[#2], GrayLevel[#3]} &@@{.9, .95, .5}
    }];

Graphics[{
    Inset[button[#], #2] & @@@ 
    Thread[{
        RandomChoice[Alphabet[], 6],
        Sqrt[3] RotationMatrix[Pi/6] . # & /@ CirclePoints[6]
    }]
}]

This has the expected bounding box problems, so I tried exportitng the rasterised images as PNG without backgrounds, but this didn't work either:

{b1, b2, b3} =
Table[
    Rasterize[
        Graphics[{
        EdgeForm[None], c, Polygon[CirclePoints[6]]
        }], 
    ImageSize -> 100],
{c, {GrayLevel[.9], GrayLevel[.95], GrayLevel[.5]}}];

Export[#2 <> ".png", #, Background -> None] & @@@ 
Thread@{{b1, b2, b3}, {"b1", "b2", "b3"}};

{b1, b2, b3} = Import[# <> ".png"] & /@ {"b1", "b2", "b3"};

Button[
    Style["A", Bold, 30], 
    Appearance -> {"Default" -> b1, "Hover" -> b2, "Pressed" -> b3}
]

Update

Thanks to @LukasLang's answer, below is the application of the button for anyone to play with:

ToExpression/@Import["https://raw.githubusercontent.com/martinq321/spellingbee/main/spellingBeeFunctions"] 

playquick[]

Answer 1

It might be easier to build your own button using EventHandler and CurrentValue:

button[letter_] := EventHandler[
   Style[{
     EdgeForm[None], Polygon[CirclePoints[6]],
     Text@Style[letter, Bold, 30]
     }, PolygonBoxOptions -> {
      BaseStyle -> Dynamic@GrayLevel@If[
          CurrentValue["MouseOver"], 
          If[CurrentValue[{"MouseButtonTest", 1}], 0.5, 0.9], 0.95
          ]
      }
    ],
   "MouseClicked" :> {}
   ];

Graphics[{Translate[button[#], #2] & @@@ 
   Thread[{RandomChoice[Alphabet[], 6], 
     Sqrt[3] RotationMatrix[Pi/6] . # & /@ CirclePoints[6]}]}]

The main trick is the use of Style[...,PolygonBoxOptions->{BaseStyle->Dynamic[...]}]: This specifies a dynamic background color without the need to reevaluate the entire graphics primitive. It also ensures that the "scope" of the CurrentValue checks is the entire button, not only the polygon: If you were to put the BaseStyle->Dynamic[...] part directly into the polygon, hovering over the letters wouldn't work, because the mouse is not over the polygon itself: