# How to determine the values of the parameters for which the system has an infinite set of the solutions? four solutions?

I mean the system

$$\begin{cases} a (x+y)+x^2-y^2=a+x-y, \\ b x y+x^2+y^2-1=0 \end{cases}$$ in $$x,y$$. Mathematica 13.1 solves it over the reals by

Reduce[{a*(x + y) + x^2 - y^2 - a - x + y == 0,  b*x*y + x^2 + y^2 - 1 == 0}, {x, y}, Reals]

(a < -3 && ((b < -2 && ((x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (-2 <= b < (-4 + 2 a^2)/
a^2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0))) || (b == (-4 + 2 a^2)/
a^2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || ((-4 + 2 a^2)/a^2 < b <
2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b ==
2 && ((x < -(1/2) - a/2 &&
y == 1 - x) || (x == -(1/2) - a/2 && (y == a + x ||
y == 1 - x)) || (-(1/2) - a/2 < x < 1/2 - a/2 &&
y == 1 - x) || (x == 1/2 - a/2 &&
y == a + x) || (x > 1/2 - a/2 && y == 1 - x))) || (2 <
b < -1 -
a && ((x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b == -1 -
a && ((x == 0 &&
y == 1) || (x ==
1 && (y == 1 + a || y == 0)) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b > -1 -
a && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))))) || (a == -3 && ((b < -2 && ((x ==
3/2 - 1/2 Sqrt[(-14 + 9 b)/(2 + b)] &&
y == -3 + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == 3/2 + 1/2 Sqrt[(-14 + 9 b)/(2 + b)] &&
y == -3 + x))) || (-2 <= b < 14/
9 && ((x == 0 && y == 1) || (x == 1 && y == 0))) || (b == 14/
9 && ((x == 0 && y == 1) || (x == 1 && y == 0) || (x == 3/2 &&
y == -(3/2)))) || (14/9 < b <
2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == 3/2 - 1/2 Sqrt[(-14 + 9 b)/(2 + b)] &&
y == -3 + x) || (x == 3/2 + 1/2 Sqrt[(-14 + 9 b)/(2 + b)] &&
y == -3 + x))) || (b ==
2 && ((x < 1 &&
y == 1 - x) || (x == 1 && (y == -2 || y == 0)) || (1 < x <
2 && y == 1 - x) || (x == 2 && y == -1) || (x > 2 &&
y == 1 - x))) || (b >
2 && ((x == 0 &&
y == 1) || (x == 3/2 - 1/2 Sqrt[(-14 + 9 b)/(2 + b)] &&
y == -3 + x) || (x == 1 &&
y == 0) || (x == 3/2 + 1/2 Sqrt[(-14 + 9 b)/(2 + b)] &&
y == -3 + x))))) || (-3 <
a < -2 && ((b < -2 && ((x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (-2 <= b < (-4 + 2 a^2)/
a^2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0))) || (b == (-4 + 2 a^2)/
a^2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || ((-4 + 2 a^2)/a^2 <
b < -1 -
a && ((x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b == -1 -
a && ((x == 0 &&
y == 1) || (x ==
1 && (y == 1 + a || y == 0)) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (-1 - a < b <
2 && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b ==
2 && ((x < -(1/2) - a/2 &&
y == 1 - x) || (x == -(1/2) - a/2 && (y == a + x ||
y == 1 - x)) || (-(1/2) - a/2 < x < 1/2 - a/2 &&
y == 1 - x) || (x == 1/2 - a/2 &&
y == a + x) || (x > 1/2 - a/2 && y == 1 - x))) || (b >
2 && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))))) || (a == -2 && ((b < -2 && ((x ==
1 - 1/2 Sqrt[(-4 + 4 b)/(2 + b)] &&
y == -2 + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == 1 + 1/2 Sqrt[(-4 + 4 b)/(2 + b)] &&
y == -2 + x))) || (-2 <= b <
1 && ((x == 0 && y == 1) || (x == 1 && y == 0))) || (b ==
1 && ((x == 0 &&
y == 1) || (x == 1 && (y == -1 || y == 0)))) || (1 < b <
2 && ((x == 0 &&
y == 1) || (x == 1 - 1/2 Sqrt[(-4 + 4 b)/(2 + b)] &&
y == -2 + x) || (x == 1 &&
y == 0) || (x == 1 + 1/2 Sqrt[(-4 + 4 b)/(2 + b)] &&
y == -2 + x))) || (b ==
2 && ((x < 1/2 &&
y == 1 - x) || (x == 1/
2 && (y == -(3/2) || y == 1/2)) || (1/2 < x < 3/2 &&
y == 1 - x) || (x == 3/2 && y == -(1/2)) || (x > 3/2 &&
y == 1 - x))) || (b >
2 && ((x == 0 &&
y == 1) || (x == 1 - 1/2 Sqrt[(-4 + 4 b)/(2 + b)] &&
y == -2 + x) || (x == 1 &&
y == 0) || (x == 1 + 1/2 Sqrt[(-4 + 4 b)/(2 + b)] &&
y == -2 + x))))) || (-2 <
a < -1 && ((b < -2 && ((x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (-2 <= b < (-4 + 2 a^2)/
a^2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0))) || (b == (-4 + 2 a^2)/
a^2 && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))) || ((-4 + 2 a^2)/a^2 <
b < -1 -
a && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))) || (b == -1 -
a && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x ==
1 && (y == 1 + a || y == 0)))) || (-1 - a < b <
2 && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b ==
2 && ((x < -(1/2) - a/2 &&
y == 1 - x) || (x == -(1/2) - a/2 && (y == a + x ||
y == 1 - x)) || (-(1/2) - a/2 < x < 1/2 - a/2 &&
y == 1 - x) || (x == 1/2 - a/2 &&
y == a + x) || (x > 1/2 - a/2 && y == 1 - x))) || (b >
2 && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))))) || (a == -1 && ((b < -2 && ((x ==
0 && (y == -1 || y == 1)) || (x == 1 &&
y == 0))) || (b == -2 && ((x < 0 &&
y == -1 + x) || (x == 0 && (y == -1 || y == 1)) || (x > 0 &&
y == -1 + x))) || (-2 < b <
2 && ((x == 0 && (y == -1 || y == 1)) || (x == 1 &&
y == 0))) || (b ==
2 && ((x < 0 &&
y == 1 - x) || (x == 0 && (y == -1 || y == 1)) || (0 < x <
1 && y == 1 - x) || (x == 1 && y == 0) || (x > 1 &&
y == 1 - x))) || (b >
2 && ((x == 0 && (y == -1 || y == 1)) || (x == 1 &&
y == 0))))) || (-1 < a <
0 && ((b < (-4 + 2 a^2)/
a^2 && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))) || (b == (-4 + 2 a^2)/
a^2 && ((x == 0 &&
y == 1) || (x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))) || ((-4 + 2 a^2)/a^2 <
b <= -2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0))) || (-2 <
b < -1 -
a && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b == -1 -
a && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == 1 && (y == 0 || y == 1 + a)))) || (-1 -
a < b < 2 && ((x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))) || (b ==
2 && ((x < -(1/2) - a/2 &&
y == 1 - x) || (x == -(1/2) - a/2 && (y == a + x ||
y == 1 - x)) || (-(1/2) - a/2 < x < 1/2 - a/2 &&
y == 1 - x) || (x == 1/2 - a/2 &&
y == a + x) || (x > 1/2 - a/2 && y == 1 - x))) || (b >
2 && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))))) || (a ==
0 && ((b <= -2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0))) || (-2 <
b < -1 && ((x == -Sqrt[(1/(2 + b))] && y == x) || (x == 0 &&
y == 1) || (x == 1 && y == 0) || (x == Sqrt[1/(2 + b)] &&
y == x))) || (b == -1 && ((x == -1 && y == -1) || (x == 0 &&
y == 1) || (x == 1 && (y == 0 || y == 1)))) || (-1 < b <
2 && ((x == -Sqrt[(1/(2 + b))] && y == x) || (x == 0 &&
y == 1) || (x == Sqrt[1/(2 + b)] && y == x) || (x == 1 &&
y == 0))) || (b ==
2 && ((x < -(1/2) &&
y == 1 - x) || (x == -(1/2) && (y == -(1/2) ||
y == 3/2)) || (-(1/2) < x < 1/2 &&
y == 1 - x) || (x == 1/2 && y == 1/2) || (x > 1/2 &&
y == 1 - x))) || (b >
2 && ((x == -Sqrt[(1/(2 + b))] && y == x) || (x == 0 &&
y == 1) || (x == Sqrt[1/(2 + b)] && y == x) || (x == 1 &&
y == 0))))) || (0 < a <
1 && ((b < (-4 + 2 a^2)/
a^2 && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0))) || (b == (-4 + 2 a^2)/
a^2 && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0))) || ((-4 + 2 a^2)/a^2 <
b <= -2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0))) || (-2 <
b < -1 -
a && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (b == -1 -
a && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == 1 && (y == 0 || y == 1 + a)))) || (-1 -
a < b < 2 && ((x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))) || (b ==
2 && ((x < -(1/2) - a/2 &&
y == 1 - x) || (x == -(1/2) - a/2 && (y == a + x ||
y == 1 - x)) || (-(1/2) - a/2 < x < 1/2 - a/2 &&
y == 1 - x) || (x == 1/2 - a/2 &&
y == a + x) || (x > 1/2 - a/2 && y == 1 - x))) || (b >
2 && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))))) || (a ==
1 && ((b < -2 && ((x == -1 && y == 0) || (x == 0 &&
y == 1) || (x == 1 && y == 0))) || (b == -2 && ((x < 1 &&
y == 1 + x) || (x == 1 && (y == 0 || y == 2)) || (x > 1 &&
y == 1 + x))) || (-2 < b <
2 && ((x == -1 && y == 0) || (x == 0 && y == 1) || (x == 1 &&
y == 0))) || (b ==
2 && ((x < -1 &&
y == 1 - x) || (x == -1 && (y == 0 || y == 2)) || (-1 < x <
0 && y == 1 - x) || (x == 0 && y == 1) || (x > 0 &&
y == 1 - x))) || (b >
2 && ((x == -1 && y == 0) || (x == 0 && y == 1) || (x == 1 &&
y == 0))))) || (a >
1 && ((b < -1 -
a && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 1 && y == 0))) || (b == -1 -
a && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 &&
y == 1) || (x == 1 && (y == 0 || y == 1 + a)))) || (-1 -
a < b < -2 && ((x == -(a/2) -
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x))) || (-2 <= b < (-4 + 2 a^2)/
a^2 && ((x == 0 && y == 1) || (x == 1 &&
y == 0))) || (b == (-4 + 2 a^2)/
a^2 && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0))) || ((-4 + 2 a^2)/a^2 < b <
2 && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 &&
y == 0))) || (b ==
2 && ((x < -(1/2) - a/2 &&
y == 1 - x) || (x == -(1/2) - a/2 && (y == a + x ||
y == 1 - x)) || (-(1/2) - a/2 < x < 1/2 - a/2 &&
y == 1 - x) || (x == 1/2 - a/2 &&
y == a + x) || (x > 1/2 - a/2 && y == 1 - x))) || (b >
2 && ((x == -(a/2) - 1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == -(a/2) +
1/2 Sqrt[(4 - 2 a^2 + a^2 b)/(2 + b)] &&
y == a + x) || (x == 0 && y == 1) || (x == 1 && y == 0)))))


As we see, the result is huge. Because of this reason it is difficult to answer the question about the number of the solutions by eye. The question is: how to do it programmatically? Maple has the RootFinding[Parametric] package to this end.

• FullSimplify does not help here. Commented Jan 5, 2023 at 16:42
• @flinty: Your edit of the title distorted the meaning of the question. I restored the original title. Commented Jan 5, 2023 at 20:32
• determinate isn't the right word - please fix it. It's an adjective, not a verb - you want "determine". Commented Jan 5, 2023 at 22:33
• @flinty: See that. Commented Jan 6, 2023 at 10:07
• that's a Ukrainian translation. There's no such thing as 'To determinate' - it's not a verb. https://dictionary.cambridge.org/dictionary/english/determinate Commented Jan 6, 2023 at 12:07

Clear["Global*"]

eqns = {a*(x + y) + x^2 - y^2 - a - x + y == 0,
b*x*y + x^2 + y^2 - 1 == 0};

sol = Solve[eqns, {x, y}, Reals];

Length@sol

(* 4 *)

eqns /. sol // Simplify


The solutions are conditionally valid.

For there to be four solutions, all of the conditions in the ConditionalExpressions must be met simultaneously.

cond = Reduce[
And @@
Cases[sol,
ConditionalExpression[_, cond_] :> cond, Infinity],
{a, b}, Reals] //
Simplify

(* ((2 + b < 0 ||
1 + a + b > 0 || (4/a^2 + b > 2 && 1 + a + b < 0)) && (a < -2 || -2 <
a < -1)) || ((4/a^2 + b < 2 ||
1 + a + b > 0 || (2 + b > 0 && 1 + a + b < 0)) && (-1 < a < 0 ||
0 < a < 1)) || (a >
1 && (4/a^2 + b > 2 || 1 + a + b < 0 || (2 + b < 0 && 1 + a + b > 0))) *)

sol2 = Assuming[cond, Solve[cond && And @@ eqns, {x, y}, Reals] //
Simplify]

(* {{x -> 0, y -> 1}, {x -> 1,
y -> 0}, {x -> 1/2 (-a - Sqrt[(4 + a^2 (-2 + b))/(2 + b)]),
y -> 1/2 (a - Sqrt[(4 + a^2 (-2 + b))/(2 + b)])}, {x ->
1/2 (-a + Sqrt[(4 + a^2 (-2 + b))/(2 + b)]),
y -> 1/2 (a + Sqrt[(4 + a^2 (-2 + b))/(2 + b)])}} *)

eqns /. sol2 // Simplify

(* {{True, True}, {True, True}, {True, True}, {True, True}} *)


The region for four solutions is

RegionPlot[cond, {a, -5, 5}, {b, -5, 5},
PlotPoints -> 75,
FrameLabel -> (Style[#, 14] & /@ {a, b})]


For example,

sol3 = Solve[eqns /. #, {x, y}] & /@
{{a -> -2, b -> -3}, {a -> 4, b -> -9/2},
{a -> -3, b -> 7/2}, {a -> 1/2, b -> 1/2}}

(* {{{x -> -1, y -> -3}, {x -> 0, y -> 1}, {x -> 1, y -> 0}, {x -> 3,
y -> 1}}, {{x -> 0, y -> 1}, {x -> 1, y -> 0}, {x -> -2 - Sqrt[10],
y -> 2 - Sqrt[10]}, {x -> -2 + Sqrt[10], y -> 2 + Sqrt[10]}}, {{x -> 0,
y -> 1}, {x -> 1, y -> 0}, {x -> 1/22 (33 - Sqrt[385]),
y -> 1/22 (-33 - Sqrt[385])}, {x -> 1/22 (33 + Sqrt[385]),
y -> 1/22 (-33 + Sqrt[385])}}, {{x -> 0, y -> 1}, {x -> 1,
y -> 0}, {x -> 1/20 (-5 - Sqrt[145]),
y -> 1/20 (5 - Sqrt[145])}, {x -> 1/20 (-5 + Sqrt[145]),
y -> 1/20 (5 + Sqrt[145])}}} *)

Length /@ sol3

(* {4, 4, 4, 4} *)

• +1. Thank you for your work. First, Solve may do non-equivalent transforms, Reduce may not. Second, can you comment you code? A good code is a commented code. Commented Jan 5, 2023 at 17:51
• BTW, Maple splits the parametric plane into 13 sets: for the interiors of 10 ones there are four solutions. Commented Jan 5, 2023 at 18:00
• Your result coincides with the Maple's result. How about the case of infinite sets of the solutions? Commented Jan 5, 2023 at 18:18
• Can you kindly explain how to find parameters for three solutions instead of four solutions? Commented Jan 5, 2023 at 22:23
• That is a significantly different question and should be posted as a separate question. Commented Jan 5, 2023 at 23:29

I'll handle dimensionality over the complexes since that's simpler, and provides an important step for finding non-generic cases (fewer than 4, or infinitely many) over the reals. Then I'll show how to start on the real solutions counting.

The point of attack is called the "discriminant variety". In relatively simple cases it can be found computationally as follows.

(1) Construct a Groebner basis for the system, treating parameters as coefficients (rather than variables).

(2) Find conditions on the parameters for which either leading coefficients, or entire polynomials, vanish.

polys = {a*(x + y) + x^2 - y^2 - a - x + y, b*x*y + x^2 + y^2 - 1};
vars = {x, y};
gb = GroebnerBasis[polys, vars, CoefficientDomain -> RationalFunctions]

(* Out[1024]= {(1 - a^2) y + (-1 + 2 a + a^2 + a b) y^2 + (-2 - 2 a - b -
a b) y^3 + (2 + b) y^4,
1 - 2 a + a^2 + b -
a b + (-1 + 2 a - a^2 - b + a b) x + (1 - a^2 + 2 a b +
a b^2) y + (-2 + 2 a - 3 b - a b - b^2 - a b^2) y^2 + (2 b +
b^2) y^3} *)


First we investigate the polynomial in y only.

lcoeffy = Coefficient[gb[[1]], y^Exponent[gb[[1]], y]];
Solve[lcoeffy == 0]

(* Out[1052]= {{b -> -2}} *)


When b = 2 the y polynomial loses a term and the number of solutions either drops or becomes infinite depending on what happens to the other polynomial.

Let's see when the entire polynomial vanishes.

allYcoeffs = CoefficientList[gb[[1]], y];
Solve[allYcoeffs == 0, {a, b}]

(* Out[1036]= {{a -> -1, b -> -2}, {a -> 1, b -> -2}} *)


When either of these are satisfied we have no conditions on y so the solution set is one dimensional (with x parametrized by y, which can take on any value) or, if the x leading term in the second basis polynomial also vanishes, all of C^2.

Next we check what happens when the coefficient for the polynomial that determines x vanishes.

lcoeffx = Coefficient[gb[[2]], x^Exponent[gb[[2]], x]];
nox = Solve[lcoeffx == 0]
Map[Solve[# == 0, y] &, gb /. nox]

(* Out[1049]= {{a -> 1}, {b -> -1 + a}}

Out[1050]= {{{y -> 0}, {y -> 1}, {y -> 1}},
{{y -> 0}, {y ->  1}, {y -> -1 + a}}} *)


We get in each case a solution for y, with x allowed to take on any value (so a one-dimensional solution set).

To take this further and count over reals, one might look into the discriminant variety of the first basis polynomial as a function of those parameters. Real solution counts can only change when solution pairs cross and, for this to happen, that variety must vanish.

ydiscrim = Discriminant[gb[[1]], y]

(* Out[1053]= (-1 + a)^4 (1 + a)^2 (2 + b) (4 - 8 a + 2 a^2 + 4 a^3 -
2 a^4 + 8 b - 8 a b - 3 a^2 b + 2 a^3 b + a^4 b + 4 b^2 -
2 a^3 b^2 + a^2 b^3) *)


So one can proceed from here by looking at the regions in real parameter space that are demarcated by this zero set.

fax =
Times @@
Rest[FactorList[(-1 + a)^4 (1 + a)^2 (2 + b) (4 - 8 a + 2 a^2 +
4 a^3 - 2 a^4 + 8 b - 8 a b - 3 a^2 b + 2 a^3 b + a^4 b +
4 b^2 - 2 a^3 b^2 + a^2 b^3)]][[All, 1]]

(* Out[1055]= (-1 + a) (1 + a) (-1 + a - b) (2 + b) (4 - 2 a^2 + a^2 b) *)

zset = ContourPlot[fax == 0, {a, -10, 10}, {b, -10, 10}]


--- edit --- I thought I should put in a better graphic.

zset = ContourPlot[fax == 0, {a, -5, 5},
{b, -5, 5}, PlotPoints -> 300]


--- end edit ---

• Seems I posted anew rather than editing. Deleted first version. Commented Jan 5, 2023 at 17:10
• +1. Nice. However , you don't completely describe the values of the parameters for which the set of the solutions is infinite. This is not only {{a -> -1, b -> -2}, {a -> 1, b -> -2}} . How about four solutions? Commented Jan 5, 2023 at 17:18
• @user64494 Re infinite solutions: It is the union of that set and the one given by Out[1049] (which tells when there is no determined value of x so it can take on any value). Commented Jan 5, 2023 at 17:52
• @user 64494 From the Groebner basis (or other general theory), there are four complex solutions, counting multiplicity, every where off the (complex) discriminant variety. That's the set where one or both leading coefficients vanish. Stated differently, neither lead coefficient vanishing implies four solutions. Commented Jan 5, 2023 at 17:54
• (1) @ does work. But if you don't use it, I won't see the message. (2) I don't actually read MSE all day. And I don't get email notifications about comments. (3) It is quite possible I made a mistake in the analysis. Wouldn't be the first time. Commented Jan 5, 2023 at 23:18

Here is my answer to the first part of the question: when the system has an infinite set of its solutions? The system {a*(x + y) + x^2 - y^2 - a - x + y == 0, b*x*y + x^2 + y^2 - 1 == 0} is equivalent to x+y-1==0&&b*x*y + x^2 + y^2 - 1 == 0||x-y+a==0&&b*x*y + x^2 + y^2 - 1 == 0. Then

Reduce[x + y - 1 == 0 && b*x*y + x^2 + y^2 - 1 == 0, {x, y}, Reals]


((b < 2 && (x == 0 || x == 1)) || b == 2 || (b > 2 && (x == 0 || x == 1))) && y == 1 - x

brings b==-2 and a - an arbitrary real number. From the second system {{a -> -1, b -> -2}, {a -> 1, b -> -2}}` is derived.