Using `Eliminate[]`:

    With[{a = {a1, a2, a3}, b = {b1, b2, b3}}, 
         Eliminate[{(a + 3 b) . (7 a - 5 b) == 0, (a - 4 b) . (7 a - 2 b) == 0,
                    cang == a . b/(Sqrt[a . a] Sqrt[b . b])}, Join[a, b]]]
       4 cang^2 == 1

    Solve[%, cang]
       {{cang -> -1/2}, {cang -> 1/2}}

Only one of the two possible solutions is valid, however:

    {(a + 3 b) . (7 a - 5 b) == 0, (a - 4 b) . (7 a - 2 b) == 0} /.
    {a -> {1, 0, 0}, b -> Append[AngleVector[2 π/3], 0]}
       {False, False}

    {(a + 3 b) . (7 a - 5 b) == 0, (a - 4 b) . (7 a - 2 b) == 0} /.
    {a -> {1, 0, 0}, b -> Append[AngleVector[π/3], 0]}
       {True, True}