mt1 = {{Cos[ξ] Cos[ϕ] - Sin[ξ] Sin[ϕ] Sin[ψ], Cos[ψ] Sin[ϕ], Cos[ϕ] Sin[ξ] + Cos[ξ] Sin[ϕ] Sin[ψ]},
           {-Cos[ξ] Sin[ϕ] - Cos[ϕ] Sin[ξ] Sin[ψ], Cos[ϕ] Cos[ψ], -Sin[ξ] Sin[ϕ] + Cos[ξ] Cos[ϕ] Sin[ψ]},
           {-Cos[ψ] Sin[ξ], -Sin[ψ], Cos[ξ] Cos[ψ]}};

    mt2 = {{a, b, c}, {b, d, e}, {c, e, h}};

Here `mt1` is an orthogonal matrix and `mt2` is a real symmetric matrix. In principle, the output of `Transpose[mt1].mt2.mt1` should be a diagonal matrix, but Mathematica cannot do it. Any suggestion is appreciated!