The following works:
First we solve the ordinaryordinary differential equation system for $x$ and $y$ up to $t=2\tau$ without considering $k$.
om = Pi; lam = 0.5; tau = 1; gam = 1;
solstart = {x[t], y[t]} /.
NDSolve[{x'[t] ==lam x[t]+om y[t],y'[t]==-om x[t]+lam y[t],
x[0] == 0.5, y[0] == 0}, {x, y}, {t, 0, 2tau}][[1]];
Then we use these solutions as history functions for the actualdelay differential equation system. The whole code:
om = Pi; lam = 0.5; tau = 1; gam = 1;
solstart = {x[t], y[t]} /.
NDSolve[{x'[t]==lam x[t]+om y[t],y'[t]==-om x[t]+lam y[t],
x[0] == 0.5, y[0] == 0}, {x, y}, {t, 0, 2}][[1]];
sol = NDSolve[{x'[t] == lam x[t] + om y[t] - k[t] (x[t] - x[t - tau]),
y'[t] == -om x[t] + lam y[t] - k[t] (y[t] - y[t - tau]),
k'[t] == gam ((x[t] - x[t - tau]) (x[t] - 2 x[t - tau] +
x[t - 2 tau]) + (y[t] - y[t - tau]) (y[t] - 2 y[t - tau] +
y[t - 2 tau])), x[t /; t <= 2tau] == solstart[[1]],
y[t /; t <= 2tau] == solstart[[2]], k[t /; t <= 2tau] == 0},
{x,y,k},{t, 0, 40}][[1]];
Plot[{x[t], y[t]} /. sol // Evaluate, {t, 0, 40}, ImageSize -> Large,
PlotRange -> {{0, 40}, {-1.5, 1.5}}]