The kernel crashed because it ran through all the stack space available to it. The Memory Management Tutorial page in the documentation states the following regarding stack space:
In the Wolfram System, one of the primary uses of stack space is in handling the calling of one Wolfram Language function by another. All such calls are explicitly recorded in the Wolfram System Stack discussed in "The Evaluation Stack". You can control the size of this stack by setting the global parameter $RecursionLimit. You should be sure that this parameter is set small enough that you do not run out of stack space on your particular computer system.
On the same page, one also finds:
If the specified stack space limit is exceeded, the program usually just exits.
So if you set $RecursionLimit too low, you get the error you refer to; if you set it too high, on the other hand, you run out of available stack space and the kernel gives up the ghost...
I'm afraid that I can't propose any general solution to your problem, unfortunately. The problem has been discussed in many venues before:
To sum up what has been said in those comments, the problem of predicting in advance how much stack your computation will need, and therefore setting a reasonable
$RecursionLimit that will prevent crashes is VERY hard, and possibly insoluble in the general case, especially using the high level code that Mathematica affords us, since this typically requires one to know how the intermediate results are stored, how big they are, where they are stored, etc.
An additional problem is the fact that it may be difficult to pinpoint exactly how much space is being used by a specific computation. In particular, functions like
MaxMemoryUsed "will not typically account for code space, stack space, or the effects of heap fragmentation" (see docs page).
Finally, it would be nice if the situation were handled more gracefully. For instance, the computation might
$Fail or be
$Aborted before crashing the kernel and losing the information / values currently stored in it, which may have taken a long while to compute.
A caveat: this is my interpretation "from the outside, looking in"; I'd love to hear from more experienced users or from those that have intimate knowledge of the inner workings of MMA.