# Function doesn't behave as expected

f[h_, y_] := y^2 + D[h[y], y]


Applying f once works fine

f[g, y]


gives

y^2 + Derivative[1][g][y]


But applying f twice doesn't work.

f[f[g, y], y]


y^2 + Derivative[1][(y^2 + Derivative[1][g][y])][y] + (2 y + (g''[y])[y]

Maybe like this?

f[h_, y_] := y^2 + D[h, y]
f[f[g[y], y], y]


2 y + y^2 + Derivative[2][g][y]

Or alternatively:

    ff[g_] := D[g, x]

ff[x^2 + h[x]]


Result: 2 x + Derivative[1][h][x]

    ff[ff[x^2 + h[x]]]


Result: 2 + (h^[Prime][Prime])[x]

Keep your original definition but add another piece to your definition to handle a second argument pattern:

ClearAll[f]
f[h_, y_] := y^2 + D[h[y], y]
f[a : _[__], y_] := y^2 + D[a, y]

f[g, y] // TeXForm


$g'(y)+y^2$

f[g[y], y] // TeXForm


$g'(y)+y^2$

f[f[g, y], y] // TeXForm (* or Nest[f[#, y] &, g, 2] *)


$g''(y)+y^2+2 y$

f[f[g[y], y], y] // TeXForm (* or Nest[f[#, y] &, g[y], 2] *)


$g''(y)+y^2+2 y$

f[f[f[g, y], y], y] // TeXForm (* or Nest[f[#, y] &, g, 3] *)


$g^{(3)}(y)+y^2+2 y+2$