How to solve this equation with matrix rank condition

Question

It is known that quadratic form $f\left(x_{1}, x_{2}\right)=x_{1}^{2}-4 x_{1} x_{2}+4 x_{2}^{2}$ can be transformed into quadratic form $g\left(y_{1}, y_{2}\right)=a y_{1}^{2}+4 x_{1} x_{2}+6 y_{2}^{2}$ by orthogonal transformation (It is known that matrix $\left(\begin{array}{ll} a & 2 \\ 2 & b \end{array}\right)$ and matrix $\left(\begin{array}{ll} 1 & -2 \\ -2 & 4 \end{array}\right)$ are congruent matrices under orthogonal transformation).

Now I want to find the value of a, b.

  Solve[Det[{{a, 2}, {2, b}}] == Det[{{1, -2}, {-2, 4}}] && 
      MatrixRank[{{a, 2}, {2, b}}] == MatrixRank[{{1, -2}, {-2, 4}}], {a, 
      b}]

But the above code returns an empty set after being run (the answer is {a->4,b->1}). What can I do to solve this matrix equation?

Updated content:

In addition, for the three-dimensional case, how to find the solution of the following matrix equation quickly:

Q = Array[x, {3, 3}]; 
A = {{1 - a, 1 + a, 0}, {1 + a, 1 - a, 0}, {0, 0, 2}} /. a -> 2; 
FindInstance[
 Thread[Transpose[Q] . A . Q == {{-4, 0, 0}, {0, 2, 0}, {0, 0, 2}}], 
 Flatten[Q], Reals]

Since Q is required to be a real matrix, the above code has been running and cannot return results.

Other examples for testing:

A = {{a, 0, 1}, {0, a, -1}, {1, -1, a - 1}};  
Transpose[Q] . A . Q == {{1, 0, 0}, {0, 1, 0}, {0, 0, 0}}

Answer 1

Not too hard to do, if you recall that one can use a rotation matrix to perform the required orthogonal similarity transformation:

{{{a, 2}, {2, b}}, TrigExpand[RotationMatrix[2 ArcTan[u]]]} /. 
 Solve[With[{rot = TrigExpand[RotationMatrix[2 ArcTan[u]]]}, 
            Flatten[Thread /@ Thread[rot.{{a, 2}, {2, b}}.Transpose[rot] ==
                                     {{1, -2}, {-2, 4}}]]],
       {a, b, u}]
   {{{{1, 2}, {2, 4}}, {{-(3/5), 4/5}, {-(4/5), -(3/5)}}},
    {{{4, 2}, {2, 1}}, {{0, 1}, {-1, 0}}},
    {{{1, 2}, {2, 4}}, {{3/5, -(4/5)}, {4/5, 3/5}}},
    {{{4, 2}, {2, 1}}, {{0, -1}, {1, 0}}}}

Note also the use of the Weierstrass substitution to ease the algebra done by Solve[].

As an example verification,

{{0, -1}, {1, 0}}.{{4, 2}, {2, 1}}.Transpose[{{0, -1}, {1, 0}}] == {{1, -2}, {-2, 4}}
   True

Answer 2

The build-in function MatrixRank cannot work in symbolic matrix. So I make a symbolic version MatrixRankSym here, then:

Reduce[Det[{{a, 2}, {2, b}}] == Det[{{1, -2}, {-2, 4}}] && 
  MatrixRankSym[{{a, 2}, {2, b}}] == MatrixRank[{{1, -2}, {-2, 4}}], {a, b}]

a!=0&&b==4/a