by hand:
\begin{align*}
\frac{1-n-ik}{1+n+ik} & =\sqrt{x}e^{iy}\\
\frac{\left( 1-n-ik\right) \left( 1+n-ik\right) }{\left( 1+n+ik\right)
\left( 1+n-ik\right) } & =\sqrt{x}\left( \cos y+i\sin y\right) \\
\frac{-k^{2}-2ik-n^{2}+1}{k^{2}+n^{2}+2n+1} & =\sqrt{x}\cos y+i\sqrt{x}\sin
y\\
i\left( \frac{-2k}{k^{2}+n^{2}+2n+1}\right) +\frac{-k^{2}-n^{2}+1}
{k^{2}+n^{2}+2n+1} & =\sqrt{x}\cos y+i\sqrt{x}\sin y
\end{align*}
Hence
\begin{align*}
\sqrt{x}\cos y & =\frac{-k^{2}-n^{2}+1}{k^{2}+n^{2}+2n+1}\\
\sqrt{x}\sin y & =\frac{-2k}{k^{2}+n^{2}+2n+1}%
\end{align*}
Use Mathematica to help solve the last part
Clear[x, y, n, k]
lhs = (1 - n - I k)/(1 + n + I k);
rhs = Sqrt[x] Exp[I y];
lhsReal = ComplexExpand[Re[lhs]];
lhsIm = ComplexExpand[Im[lhs]];
rhsReal = ComplexExpand[Re[rhs]];
rhsIm = ComplexExpand[Im[rhs]];
eq1 = Assuming[Element[{x, y}, Reals] && x > 0 && y > 0, Simplify[lhsReal == rhsReal]];
eq2 = Assuming[Element[{x, y}, Reals] && x > 0 && y > 0, Simplify[lhsIm == rhsIm]];
sol=Solve[{eq1, eq2}, {n, k}]
Update:
let me simplify the solution so verify it is the same solution obtained by the nice method below by b.gatessucks by applying Simplify :
Simplify[k /. sol]
Simplify[n /. sol]