First define functions to characterize your Polygon R:
f1[x_] = 2 Pi/Sqrt[3]/(-2 Pi/3 - (-4 Pi/3)) (x + 4 Pi/3)
f2[x_] = -2 Pi/Sqrt[3]/(-2 Pi/3 - (-4 Pi/3)) (x + 4 Pi/3)
f3[x_] = 2 Pi/Sqrt[3]
f4[x_] = -2 Pi/Sqrt[3]
f5[x_] = -2 Pi/Sqrt[3]/(-2 Pi/3 - (-4 Pi/3)) (x - 4 Pi/3)
f6[x_] = 2 Pi/Sqrt[3]/(-2 Pi/3 - (-4 Pi/3)) (x - 4 Pi/3)
func1[x_] =
Piecewise[{{f1[x], -4 Pi/3 <= x <= -2 Pi/3}, {f3[x], -2 Pi/3 < x <
2 Pi/3}, {f5[x], 2 Pi/3 <= x <= 4 Pi/3}}, 0]
func2[x_] =
Piecewise[{{f2[x], -4 Pi/3 <= x <= -2 Pi/3}, {f4[x], -2 Pi/3 < x <
2 Pi/3}, {f6[x], 2 Pi/3 <= x <= 4 Pi/3}}, 0]
Test it
Plot[{func1[x], func2[x]}, {x, -5, 5}, Epilog -> Point[Identity @@ R]]
Define your integration and FindRoot for m with given U:
f[x_, y_] =
3 + 2 Cos[Sqrt[3]/2 y + x/2] + 2 Cos[Sqrt[3]/2 y - x/2] + 2 Cos[x]
int[U_?NumericQ, m_?NumericQ] :=
NIntegrate[
1/Sqrt[U^2 m^2 + f[x, y]], {x, -4 Pi/3, 4 Pi/3}, {y, func2[x],
func1[x]}, MaxRecursion -> 100] // Quiet
mfr[U_] :=
m /. FindRoot[A U int[U, m] == 1, {m, 0, 10}, MaxIterations -> 2000]
mfrneg[U_] :=
m /. FindRoot[A U int[U, m] == 1, {m, -10, 0}, MaxIterations -> 2000]
With Plot you see, that a positive and a negitive m fullfil the condition, but only for U > 0.371841
Manipulate[
Plot[{1, A U int[U, m]}, {m, -6, 6}, PlotRange -> {0, 3}], {U, 10,
0}]
FindRoot[A U int[U, 0] == 1, {U, 0, 1},
MaxIterations -> 2000]
(* {U -> 0.371841} *)
Plot the two m as a function of U
pl = Plot[{mfr[U], mfrneg[U]}, {U, .4, 10},
PlotRange -> {{0, 10}, {-3.5, 3.5}}]
