The following works:

First we solve the ordinary differential equation system for $x$ and $y$ up to $t=2\tau$ without considering $k$. 

    om = Pi; lam = 0.5; tau = 1; gam = 1;
    solstart = {x[t], y[t]} /. 
                NDSolve[{x'[t] ==lam x[t]+om y[t],y'[t]==-om x[t]+lam y[t],
                x[0] == 0.5, y[0] == 0}, {x, y}, {t, 0, 2tau}][[1]];

Then we use these solutions as history functions for the actual system.
The whole code:

    om = Pi; lam = 0.5; tau = 1; gam = 1;
    solstart = {x[t], y[t]} /. 
                NDSolve[{x'[t]==lam x[t]+om y[t],y'[t]==-om x[t]+lam y[t],
             x[0] == 0.5, y[0] == 0}, {x, y}, {t, 0, 2}][[1]];

    sol = NDSolve[{x'[t] == lam x[t] + om y[t] - k[t] (x[t] - x[t - tau]),
                   y'[t] == -om x[t] + lam y[t] - k[t] (y[t] - y[t - tau]), 
                   k'[t] == gam ((x[t] - x[t - tau]) (x[t] - 2 x[t - tau] + 
             x[t - 2 tau]) + (y[t] - y[t - tau]) (y[t] - 2 y[t - tau] +
             y[t - 2 tau])), x[t /; t <= 2tau] == solstart[[1]], 
             y[t /; t <= 2tau] == solstart[[2]], k[t /; t <= 2tau] == 0},
             {x,y,k},{t, 0, 40}][[1]];

    Plot[{x[t], y[t]} /. sol // Evaluate, {t, 0, 40}, ImageSize -> Large, 
          PlotRange -> {{0, 40}, {-1.5, 1.5}}]