This is a direct descendant of two other recent questions, 3D and Equivalence, both of which have been answered in skillful, interesting manners. (See also the comment [actually answer] of JimB to 3D5D, applying the methodology of 3D to a higher-order problem.)
We would now like to turn the focus from the 3D constrained integration problem posed at the very outset in 3D to a 2D one, by modifying the constraint (for absolute separability) there
Boole[Subscript[λ, 1] > Subscript[λ, 2] && Subscript[λ, 2] > Subscript[λ, 3] && Subscript[λ, 3] > 1 - Subscript[λ, 1] - Subscript[λ, 2] - Subscript[λ, 3] && Subscript[λ, 1] - Subscript[λ, 3] < 2 Sqrt[Subscript[λ, 2] (1 - Subscript[λ, 1] - Subscript[λ, 2] - Subscript[λ, 3])]]
so that the inequality
Subscript[λ, 1] - Subscript[λ, 3] < 2 Sqrt[Subscript[λ, 2] (1 - Subscript[λ, 1] Subscript[λ, 2] - Subscript[λ, 3]
becomes an equality
Subscript[λ, 1] - Subscript[λ, 3] ==2 Sqrt[Subscript[λ, 2] (1 - Subscript[λ, 1] Subscript[λ, 2] - Subscript[λ, 3]
The resultant 2D integration (formulation given at end of question) should yield the Hilbert-Schmidt probability that a "two-qubit" state lies on the boundary of the absolutely separable states, rather than as in the 3D formulation within the volume of such states.
Now, relatedly, in eq. (35) of 2009paper the inverse trigonometric function based formula ($\approx 20.9648519$)
-((3840 (-5358569267936 + 33756573946095 Sqrt[2] [Pi] - 270052591568760 Sqrt[2] ArcCot[Sqrt[2]] + 11149704525960 Sqrt[2] ArcCot[2 Sqrt[2]] + 270052591568760 Sqrt[2] ArcCot[3 + Sqrt[2]]))/(-1959684729929728 + 1601255307608064 Sqrt[2] + 1529087492782080 Sqrt[2] [Pi] - 45247615492565918250 Sqrt[2] ArcCot[Sqrt[2]] + 22619730179635540245 Sqrt[2] ArcSec[3]))
was given for the Hilbert-Schmidt area-volume ratio of the two-qubit absolutely separable states.
Now, I would like to ask, first, whether this formula can be condensed/simplified, possibly along the lines of that employed by the user yarchik in the answer to Equivalence, in which the FindIntegerNullVector command was employed.
Secondly, I would like to ask if the area-to-volume ratio formula could itself be rederived by solving the 2D constrained integration problem indicated at the beginning of this question--followed by the scaling of its result by the answer of JimB
29902415923/497664 - 50274109/(512 Sqrt[2]) - (3072529845 π)/(32768 Sqrt[2]) +(1024176615 ArcCos[1/3])/(4096 Sqrt[2])
given in 3D for the absolute separability Hilbert-Schmidt probability for the two-qubit states.
We must note, though, that the original 3D constrained integration problem posed in 3D was solved there in an unconstrained form, employing a change-of-variables
change = {Subscript[λ, 1] -> x/(1 + 2 x), Subscript[λ, 2] -> y/(1 + y) (1 + x)/(1 + 2 x), Subscript[λ, 3] -> z 1/(1 + y) (1 + x)/(1 + 2 x)};
just recently provided by N. Tessore. This transformed the problem into
Integrate[integrand2, {z, 1/2, 1}, {y, z, 2 + 2 Sqrt[1 - z] - z}, {x, y, 2 Sqrt[-((-y - 2 y^2 - y^3 + y z + 2 y^2 z + y^3 z)/(-1 + y + z)^4)] + ( 4 y + z - 3 y z - z^2)/(-1 + y + z)^2}],
where
integrand2 = (9081072000 (1 + x)^8 (x - y)^2 (1 - 2 z)^2 (y - z)^2 (-1 + y + z)^2 (z + x (-1 - y + z))^2 (-1 + z + x (y + z))^2)/((1 + 2 x)^16 (1 + y)^15) .
The question of what would be a suitable change-of-variables in the requested dimension reduction scenario seems of interest.
In his comment to the originally-posed question, JimB simplified (LeafCount 55 vs. 96) the area-volume ratio given above to
-((15 (-1339642316984 + 1393713065745 Sqrt[2] \[Pi] -
2787426131490 Sqrt[2] ArcCos[1/3]))/(2 (-956877309536 + 781862943168 Sqrt[2] + 746624752335 Sqrt[2] \[Pi] - 1990999339560 Sqrt[2] ArcCos[1/3])))
As noted in my comment in response to that of JimB, I observed that FullSimplify applied to the product of this result and the earlier one of his (given above)
`29902415923/497664 - 50274109/(512 Sqrt[2]) - (3072529845 π)/(32768 Sqrt[2]) +(1024176615 ArcCos[1/3])/(4096 Sqrt[2])`
gives us simply
(5 (-669821158492 + 1393713065745 Sqrt[2] ArcCot[2 Sqrt[2]]))/5308416 ≈0.0766949
which I now conjecture should be the result of the requested 2D integration. This integration problem is expressible as ({Subscript[[Lambda], 1] -> x, Subscript[[Lambda], 2] -> y, Subscript[[Lambda], 3] -> z}) in the form
Integrate[9081072000 (x - y)^2 (1 - 3 x + 3 y - 4 Sqrt[y - 2 x y])^2 (1 - 3 x +
y - 2 Sqrt[y - 2 x y])^2 (2 y - 2 Sqrt[y - 2 x y])^2 (-1 + 2 x +
2 Sqrt[y - 2 x y])^2 (x - 3 y + 2 Sqrt[y - 2 x y])^2 Boole[x > y && 3 y > x + 2 Sqrt[y - 2 x y] &&
3 x + 4 Sqrt[y - 2 x y] > 1 + 3 y], {y, 0, 1}, {x, 0, 1}].
The transformation {z -> x - 2 y + 2 Sqrt[y - 2 x y]} was used to reduce the original 3D problem to the 2D one.
In a series of three consecutive comments to the answer (956877309536 + 243 Sqrt[2] (-3217542976 + 1024176615 ArcCos[5983/6561])))/2654208 given by yarchik, I indicated that the alternative (seemingly superior) use of the transformation {y -> 1/2 (1 - x - z + Sqrt[1 - 2 x - 2 z + 4 x z])}
yields a result
(5 (-1339642316984 + 1393713065745 Sqrt[2] ArcTan[(4 Sqrt[2])/7]))/5308416
apparently exactly twice the conjecture (5 (-669821158492 + 1393713065745 Sqrt[2] ArcCot[2 Sqrt[2]]))/5308416 ≈0.0766949 stated in the question, based on eq. (35) in the cited 2009 paper and an earlier result of JimB in this context.
To re-emphasize, the motivation behind the use of the transformations was to convert the absolute separability probability inequality into an equality.
{{ArcCot[2 Sqrt[2]], ArcCot[3 + Sqrt[2]], ArcCot[Sqrt[2]]}can be written as linear combinations of $\pi$ and $\phi=\cos ^{-1}\left(\frac{1}{3}\right)$ resulting in $-\frac{15 \left(-2787426131490 \sqrt{2} \phi +1393713065745 \pi \sqrt{2}-1339642316984\right)}{2 \left(-1990999339560 \sqrt{2} \phi +746624752335 \pi \sqrt{2}+781862943168 \sqrt{2}-956877309536\right)}$. $\endgroup$