For this one can just have replacement rules that force all products (including squares) within a variable class to be zero.
xvars = Array[x, 4];
yvars = Array[y, 3];
xrels = Union[Flatten[Outer[Times, xvars, xvars]]];
yrels = Union[Flatten[Outer[Times, yvars, yvars]]];
vars = Join[xvars, yvars];
rels = Join[xrels, yrels];
To obtain a nontrivial example (one that won't give all zeros at the end), we'll use random polynomials that tend to weight the monomial powers in a way that favors lower degrees.
randpoly[vars_, len_] :=
Module[{vlen = Length[vars]},
RandomInteger[{-10, 10}, len] .
Table[Apply[Times,
RandomChoice[vars,
RandomChoice[Reverse[Range[vlen]] -> Range[vlen]]]], {len}]]
Now create a 3 x 3 matrix with polynomials of length up to 8 (could be less, if there are repeated terms).
SeedRandom[1234];
mat = Table[randpoly[Join[xvars, yvars], 8],
{3}, {3}]
(* {{-6 x[2]^2 x[3] + 7 x[1] x[3]^2 + 10 x[2] x[4]^2 + 10 x[4] y[1]^2 -
18 y[2] + 6 x[2] y[3] - 10 x[1] y[1] y[3], -10 x[1] +
9 x[1] y[1] - 8 x[3] x[4] y[1] - 3 x[4]^2 y[2] - 9 y[3] -
2 x[2] x[3] y[3] - 3 x[1] x[3] y[1] y[2]^2 y[3],
2 x[2] + 10 x[3]^2 - 10 x[1] y[1]^4 + 9 x[1] x[2]^2 y[1] y[2] -
6 x[4]^2 y[1] y[2] - 5 y[3] -
3 x[2] x[3] y[1] y[3]}, {10 x[1]^3 x[2] - 4 x[3]^2 x[4] -
8 x[3] x[4]^2 - y[1] + 5 y[2] + 3 y[3] - x[3] y[3] +
y[1] y[2] y[3], -2 y[1] + 4 x[2]^2 y[1]^3 + 6 x[1] x[4] y[1]^3 -
8 x[4] y[2] + 2 x[2] x[3] y[2]^2 + 7 x[1]^2 x[2] x[4] y[3] -
y[3]^2 - 4 x[1] x[4]^2 y[1] y[3]^3,
5 x[2] + 10 x[1] x[3] - 5 x[1]^2 x[4] + 6 x[3] x[4] - 9 x[2] y[1] -
8 x[3] y[1]^2 - 6 x[2] y[3] + 8 x[1] x[3] y[3]}, {-6 x[3] +
10 x[4] + 5 y[3] + 7 x[1] y[3] - 6 x[3] y[3] - x[1] y[1] y[3] +
10 x[2] x[3] y[1] y[3] + 4 x[1]^2 y[1]^2 y[3]^2,
4 x[1] + 6 x[3] + 10 x[1] x[3] - 3 x[1] x[3]^2 y[1] - 7 y[2] -
x[3]^2 x[4] y[2]^2 - 10 x[2] y[3] + 8 x[1] x[3] x[4]^3 y[3],
9 x[2] - 7 x[4] + 10 y[1] - 5 x[3]^2 y[1] - 9 x[2] y[1]^2 +
6 x[1] x[2] x[4] y[2] + 2 x[4] y[3] - 2 x[4] y[1] y[3]}} *)
We use polynomial reduction by the relations to get rid of all bad products and powers. This will leave us with linear terms and quadratics that are products of a single variable from each class.
redmat0 = PolynomialReduce[mat, rels, vars]
[[All, All, 2]]
(* Out[180]= {{-18 y[2] + 6 x[2] y[3], -10 x[1] + 9 x[1] y[1] - 9 y[3],
2 x[2] - 5 y[3]}, {-y[1] + 5 y[2] + 3 y[3] - x[3] y[3], -2 y[1] -
8 x[4] y[2],
5 x[2] - 9 x[2] y[1] - 6 x[2] y[3]}, {-6 x[3] + 10 x[4] + 5 y[3] +
7 x[1] y[3] - 6 x[3] y[3], 4 x[1] + 6 x[3] - 7 y[2] - 10 x[2] y[3],
9 x[2] - 7 x[4] + 10 y[1] + 2 x[4] y[3]}} *)
We use a standard tactic to keep the quadratics and remove the linear terms.
redmat = redmat0 /. Thread[vars -> t*vars] /.
t^2 -> 1 /. t -> 0
(* Out[181]= {{6 x[2] y[3], 9 x[1] y[1],
0}, {-x[3] y[3], -8 x[4] y[2], -9 x[2] y[1] -
6 x[2] y[3]}, {7 x[1] y[3] - 6 x[3] y[3], -10 x[2] y[3],
2 x[4] y[3]}} *)