Return to Revisions

Using Eliminate[]:

With[{a = {a1, a2, a3}, b = {b1, b2, b3}}, 
     Eliminate[{(a + 3 b) . (7 a - 5 b) == 0, (a - 4 b) . (7 a - 2 b) == 0,
                cang == a . b/(Sqrt[a . a] Sqrt[b . b])}, Join[a, b]]]
   4 cang^2 == 1

Solve[%, cang]
   {{cang -> -1/2}, {cang -> 1/2}}

Only one of the two possible solutions is valid, however:

{(a + 3 b) . (7 a - 5 b) == 0, (a - 4 b) . (7 a - 2 b) == 0} /.
{a -> {1, 0, 0}, b -> Append[AngleVector[2 π/3], 0]}
   {False, False}

{(a + 3 b) . (7 a - 5 b) == 0, (a - 4 b) . (7 a - 2 b) == 0} /.
{a -> {1, 0, 0}, b -> Append[AngleVector[π/3], 0]}
   {True, True}