Beginning with
D[X[t, x], t] + D[Y[t, x], t] + X D[Y[t, x], x] + X Y == 0
First replace the derivative by some other function to simplify manipulations.
%%[[1]] /. Derivative[z1__][z2__][z3__] -> W[{z1}, z2, {z3}]
Make the first order substitution.
%/. {X -> X0 + d X1, Y -> Y0 + d Y1}
Expand the second argument of W. (Depending on the complexity of the second argument, more transformations may be needed.)
%/. W[z__] :> Thread[W[z], Plus] /. W[z1__, d z2__, z3__] :> d W[z1, z2, z3]
Perform Series and extract the Coefficient of d.
Coefficient[Normal[Series[%, {d, 0, 1}]], d]
Finally, replace W by Derivative
(% /. W[{z1__}, z2_, {z3__}] :> Derivative[z1][z2][z3]) == 0
(* X1*Y0 + X0*Y1 + X1*Derivative[0, 1][Y0][t, x] + X0*Derivative[0, 1][Y1][t, x]
+ Derivative[1, 0][X1][t, x] + Derivative[1, 0][Y1][t, x] == 0 *)
This process is laborious because such functions as Expand do not work with Derivative. Perhaps, in a future release of Mathematica.
Addendum
Note that a much simpler approach is available, if the equation to be linearized exists in an Unevaluated form.
Unevaluated[(D[X[t, x], t] + D[Y[t, x], t] + X[t, x] D[Y[t, x], x] + X[t, x] Y[t, x] == 0)] /.
{X[t, x] -> X0[t, x] + d X1[t, x], Y[t, x] -> Y0[t, x] + d Y1[t, x]};
Coefficient[Normal[Series[%[[1]], {d, 0, 1}]], d] == 0
yields the desired result.