How to find the particular solution in symbolic form of an overdetermined system of equations

Question

I want to use mathematica to reproduce a conclusion in a paper, which is solve an overdetermined system of equations, $$ \begin{array}{l} c^{2}(n-1)^{2}\left(a_{0}+a_{1} Y+a_{2} Y^{2}\right)^{3}+a(n-1)^{2}\left(a_{0}+a_{1} Y+a_{2} Y^{2}\right)^{2}+b n(2 n-1)\left(1-Y^{2}\right)^{2}\left(a_{1}+2 a_{2} Y\right)^{2} \\ -b n(n-1)\left(a_{0}+a_{1} Y+a_{2} Y^{2}\right)\left(1-Y^{2}\right)\left(-2 Y\left(a_{1}+2 a_{2} Y\right)+\left(1-Y^{2}\right) 2 a_{2}\right)=0 . \end{array} $$

Expanding the above expression into a polynomial in terms of $Y$ and equating the coefficients of all powers of $Y$ to zero yields an overdetermined system of equations.

The equations is quite lengthy, so I will list it in the end, but it involves natural number $n$ greater than 1 and real constants $a,b,c$. The variables to solve for are $a_0, a_1,a_2$.

The conclusion is,

When $\frac{a}{b}=-\frac{4n^{2}}{\left(n-1\right)^{2}}$, we can get two sets of solutions, $$ \begin{aligned} &1)\quad a_0=-\frac{a\left(n+1\right)}{2c^2n},\quad a_1=0,\quad a_2=\frac{a\left(n+1\right)}{2c^2n},\quad c=c,\\ &2)\quad a_0=\frac{2bn\left(n+1\right)}{c^2\left(n-1\right)^2},\quad a_1=0,\quad a_2=-\frac{2bn\left(n+1\right)}{c^2\left(n-1\right)^2},\quad c=c, \end{aligned} $$

Here is my attempt,

(* This part of the code is for deriving the equation, so you can ignore it. *)
eqn = c^2*(n - 1)^2*S^3 + a*(n - 1)^2*S^2 + 
b*n*(2*n - 1)*(1 - Y^2)^2*(DS)^2 - 
b*n*(n - 1)*S*(1 - Y^2)*(-2*Y*DS + (1 - Y^2)*D2S);
    
S[Y_] := a0 + a1*Y + a2*Y^2;
    
DS = D[S[Y], Y];
D2S = D[DS, Y];


(* Here, we can obtain the formula mentioned above *)
seqn = eqn /. {S -> a0 + a1*Y + a2*Y^2, DS -> DS, D2S -> D2S}

(* get the overdetermined system of equations *)
coeList = CoefficientList[Expand[seqn], Y];
equList = Thread[coeList == 0];
equList
    
(* in this way it produces two sets of trivial solutions *)
solutions = Solve[equList, {a0, a1, a2}];
solutions

In fact, I attempted to manually derive the condition $\frac{a}{b}=-\frac{4n^{2}}{\left(n-1\right)^{2}}$ but couldn't find it. I also tried solving the system by using it as a condition(already known) and use Reduce but the answer solves $n$ directly, which is what I don’t want.

solutions = 
  ToRules[Reduce[
    Append[equList, a/b == -4*n^2/(n - 1)^2], {a0, a1, a2}]];
solutions

(* Here is the output *)
Sequence[{n -> 0, c -> 0, a -> 0}, 
{n -> 0, c -> 0, a -> 0, a1 -> 0}, 
{n -> 0, c -> 0, a -> 0, a2 -> 0}, 
{n -> 0, c -> 0, a -> 0, a1 -> 0, a2 -> 0}, 
{a -> -((4 b n^2)/(-1 + n)^2), a0 -> 0, a1 -> 0, a2 -> 0}, 
{a -> -((4 b n^2)/(-1 + n)^2), a0 -> (-3 a^2 + 13 a b + a^2 n - 23 a b n + 52 b^2 n^2 + 12 b^2 n^3)/(2 c^2 (a + 4 b n^2 + 3 b n^3)), a1 -> 0, 
  a2 -> (3 a - a n - 4 b n)/(2 c^2)}, 
{c -> 0, a -> -((4 b n^2)/(-1 + n)^2), a0 -> 0, a1 -> 0, a2 -> 0}, {n -> -1, 
  a -> -b, a0 -> 0, a1 -> 0, a2 -> 0}, {n -> -1, a -> -b, a0 -> b/c^2,
   a1 -> 0, a2 -> 0}, {n -> -1, c -> 0, a -> -b, a0 -> 0, a1 -> 0, 
  a2 -> 0}, {n -> -1, c -> 0, a -> -b, a1 -> 0, a2 -> -a0}, {n -> 0, 
  a -> 0, a0 -> 0, a1 -> 0, a2 -> 0}, {n -> 5/3, a -> -25 b, 
  a0 -> (20 b)/c^2, a1 -> 0, 
  a2 -> -a0}, {a -> -((4 b n^2)/(-1 + n)^2), a0 -> 0, a1 -> 0, 
  a2 -> 0}, {a -> -((4 b n^2)/(-1 + n)^2), a0 -> -(a/c^2), a1 -> 0, 
  a2 -> 0}]

So my question is,

1. could the condition $\frac{a}{b}=-\frac{4n^{2}}{\left(n-1\right)^{2}}$ be found in using Mathematica?
2. How to solve this overdetermined system of equations without solving $n$ directly.

Any help would be greatly appreciated!

---

The "big" overdetermined system of equations,by the way, the author of the paper use Maple to solve this but did not give the code. $$ \begin{aligned} Y^{0}: & 2 b n^{2} a_{1}^{2}-b n a_{1}^{2}+c^{2} a_{0}^{3}+a a_{0}^{2}-2 n c^{2} a_{0}^{3}-2 n a a_{0}^{2}+n^{2} c^{2} a_{0}^{3} \\ & +a n^{2} a_{0}^{2}-2 b n^{2} a_{0} a_{2}+2 b n a_{0} a_{2}=0, \\ & 2 b n^{2} a_{0} a_{1}-4 a n a_{0} a_{1}-2 b n a_{1} a_{2}+3 c^{2} a_{0}^{2} a_{1}-6 c^{2} a_{0}^{2} a_{1} n+6 b n^{2} a_{1} a_{2} \\ Y^{1}: & 3 c^{2} a_{0}^{2} a_{1} n^{2}-2 b n a_{1} a_{0}+2 a n^{2} a_{0} a_{1}+2 a a_{0} a_{1}=0, \\ & a a_{1}^{2}-6 c^{2} n a_{0}^{2} a_{2}+3 c^{2} n^{2} a_{0} a_{1}^{2}+3 c^{2} a_{0} a_{1}^{2}-2 b n a_{2}^{2}-8 b n a_{0} a_{2}+2 a a_{0} a_{2}-4 a n a_{0} a_{2} \\ Y^{2}: & -2 a n a_{1}^{2}+8 b n^{2} a_{0} a_{2}+a n^{2} a_{1}^{2}-2 b n^{2} a_{1}^{2}-6 c^{2} n a_{0} a_{1}^{2}+6 b n^{2} a_{2}^{2}+3 c^{2} n^{2} a_{0}^{2} a_{2}+3 c^{2} a_{0}^{2} a_{2} \\ & +2 a n^{2} a_{0} a_{2}=0,\\ Y^{3}: & 6 c^{2} n^{2} a_{0} a_{1} a_{2}+2 a a_{1} a_{2}+2 b n a_{0} a_{1}+6 c^{2} a_{0} a_{1} a_{2}-2 b n a_{1} a_{2}-2 c^{2} n a_{1}^{3}-2 b n^{2} a_{0} a_{1} \\ & -6 b n^{2} a_{1} a_{2}-4 a n a_{1} a_{2}+c^{2} a_{1}^{3}+2 a n^{2} a_{1} a_{2}-12 c^{2} n a_{0} a_{1} a_{2}+c^{2} n^{2} a_{1}^{3}=0, \\ Y^{4}: & a n^{2} a_{2}^{2}+3 c^{2} a_{1}^{2} a_{2}+3 c^{2} n^{2} a_{2}^{2} a_{0}-8 b n^{2} a_{2}^{2}+6 b n a_{0} a_{2}+b n a_{1}^{2}-2 a a_{2}^{2} n+3 c^{2} a_{2}^{2} a_{0} \\ & -6 c^{2} a_{1}^{2} a_{2} n+a a_{2}^{2}+3 c^{2} a_{1}^{2} n^{2} a_{2}-6 b n^{2} a_{0} a_{2}-6 c^{2} a_{2}^{2} a_{0} n=0, \\ Y^{5}: & 4 b n a_{1} a_{2}+3 c^{2} n^{2} a_{1} a_{2}^{2}-6 c^{2} n a_{1} a_{2}^{2}+3 c^{2} a_{1} a_{2}^{2}=0, \\ Y^{6}: & 2 b n^{2} a_{2}^{2}+2 b n a_{2}^{2}+c^{2} a_{2}^{3}+c^{2} a_{2}^{3} n^{2}-2 c^{2} a_{2}^{3} n=0 . \end{aligned} $$

Answer 1

Try

eqn = c^2*(n - 1)^2*S[Y]^3 + a*(n - 1)^2*S[Y]^2 + 
   b*n*(2*n - 1)*(1 - Y^2)^2*(S'[Y])^2 - 
   b*n*(n - 1)*S[Y]*(1 - Y^2)*(-2*Y*S'[Y] + (1 - Y^2)*S''[Y]) /. 
  S -> Function[Y, a0 + a1*Y + a2*Y^2]

cond=Reduce[0 == CoefficientList[eqn, Y] , {a0, a1, a2 }] //Simplify[#, {Element[n, Integers], n > 1}] &
(*(a == 0 && c == 0 && ((a1 == 0 && a2 == 0) || b == 0)) || 
(a1 ==0 && ((c != 0 && ((a == -((4 b n^2)/(-1 + n)^2) && 
a0 == -(((1 + n) (a (-1 + n)^2 - 8 b n^2))/(6 c^2 (-1 + n)^2 n)) &&
a2 == (a0 (a + a0 c^2) (-1 + n))/(2 b n) && a0 != 0 &&b != 0) 
|| (a0 == -(a/c^2) && a2 == 0))) || (a0 == 0 &&a2 == 0)))*)

Perhaps ToRules helps to create a more easily readable result

ToRules[cond]  
(*Sequence[
{a -> 0, c -> 0, a1 -> 0, a2 -> 0},
{a -> 0, c -> 0,b -> 0}, 
{a1 -> 0, a -> -((4 b n^2)/(-1 + n)^2),a0 -> -(((1 + n) (a (-1 +n)^2 - 8 b n^2))/(6 c^2 (-1 + n)^2 n)), a2 -> (a0 (a + a0 c^2) (-1 + n))/(2 bn)}, 
{a1 -> 0, a0 -> -(a/c^2),a2 -> 0}, 
{a1 -> 0, a0 -> 0, a2 -> 0}]*)