Here's a simple series:

Sum[t^k DiscreteDelta[k]/k!,{k,0,Infinity}]

Mathematica says that it doesn't converge. But replacing Infinity by any positive integer I get 1. The same is for the case when I remove k! or t^k from the expression.

What's going on? Shouldn't k! improve convergence rather than lead to divergence?

Note also that Mathematica can solve this easily:

Sum[f[k] DiscreteDelta[k], {k, 0, Infinity}]
(* f[0] *)