I've never known there to be a solution to a hanging Front End. With new versions comes new functionality, so maybe there is or there will be.
Before I address the Front End, you can stop the Kernel from sending more updates to the Front End in several ways:
- Menu Evaluation > Abort Evaluation. (Front End must be responsive.)
- Menu Evaluation > Quit Kernel > Local. (Front End must be responsive.)
- Use your computers process manager or activity monitor to kill the Kernel.
Generally, the Front End will continue running. (I have had the Front End crash when I kill the Kernel in older versions, but this is less common with recent versions.) However, the Front End will continue to process input it has already received from the Kernel, and this could potentially be a very large amount of input.
There is no Front End "Abort Notebook Update" menu command.
One reason why I expect never to see such function:
If the Front End is nonresponsive, it cannot actually receive any input event. The menus do not work. The "Abort Evaluation" keyboard shortcut does not work. Sometimes the OS lets you queue an input, but it won't have an effect until the Front End receives and processes it. I have had to wait a few minutes before the Front End gets to it, which will be after it finishes whatever task(s) it was handling before I tried to abort. It's probably worth it to send feedback to WRI to ask them to improve the behavior. For instance, the Kernel could put up a dialog before sending a potentially overwhelming output to the Front End.
A reason why one might expect to see it at some point:
A notebook has a certain syntax. If a file has the wrong syntax, then it is likely Mathematica will not recognize it as a notebook. Perhaps it can do some repair. For instance, the "AuthorTools`" package has the NotebookRestore[] palette, which is irrelevant to the task at hand. However, the point is that once the Front End starts making a change to a notebook, it cannot stop — unless it can revert to the original upon an abort signal. Since we have "undo," one feels this should be possible. Those of us who saw how long it took to work out the intricacies of notebook-undo may feel this is easier said than done. It also assumes the first problem can be solved.
Workarounds?
Each of the following suggestions has to be put in place before you have made the mistake of asking for too much output. Each handles the case of a single output that is too large. Neither handles the case of too many small outputs. I don't have a suggest for that. I should disclose something about the second suggestion, which uses $Post. You will see it, $Pre, $PrePrint used in some answers. I pretty much never use them myself, except maybe in a one-off execution. Ineveitably, something will come up, and when I get back to Mathematica I would have forgotten that I set it. Then occasionally confusing things will happen because of it, and it will take me a long time to realize what's wrong.
1. The system has a built-in check on the size of outputs, namely, $OutputSizeLimit, one megabyte by default. You could make it smaller. The output is still processed by the Front End and displayed in a shortened form.
2. To avoid processing of the output by the Front End, intercept the output with $Post and send it on only if it's small. The code below uses $OutputSizeLimit, but you could use your own variable to set a different limit. One might also consider using LeafCount[#] > $myLeafCountLimit for a limit, after setting a convenient value for $myLeafCountLimit.
Clear[$myLargeOutputs];
$myLargeOutputs::toobig =
"Output too big. Saved in $myLargeOutputs[``]. Be sure to execute Unset[$myLargeOutputs[``]] or Clear[$myLargeOutputs] after you are done with it to free memory.";
$Post = If[TrueQ[ByteCount[#] > $OutputSizeLimit],
Message[$myLargeOutputs::toobig, $Line, $Line];
$myLargeOutputs[$Line] = #;
With[{line = $Line}, Defer[$myLargeOutputs[line]]],
#] &;
RandomInteger[1, 10^6]
$myLargeOutputs::toobig: Output too big. Saved in $myLargeOutputs[4]. Be sure to execute Unset[$myLargeOutputs[4]] or Clear[$myLargeOutputs] after you are done with it to free memory.
(* $myLargeOutputs[4] *)
Should you want to inspect the expression anyway, the following gets around $Post, since the return value of CellPrint[] is Null after printing a cell to the evaluation notebook. One could also use Print[Short[$myLargeOutputs[4]]].
CellPrint[ExpressionCell[Short[$myLargeOutputs[4]], "Output"]]
Another way to see the output is to raise the $OutputSizeLimit:
$OutputSizeLimit = Infinity;
Short[$myLargeOutputs[4]]
$OutputSizeLimit = 2^20; (* reset to 1MB *)
If you accumulate many large outputs, you may have memory problems eventually.
myResult = $myLargeOutput[4]; (* save one or more outputs (optional) *)
Clear[$myLargeOutput] (* clear the rest *)
3. This third suggestion seems simpler but has some drawbacks. The following controls how the output is printed to the notebook, but the output is processed by the Front End. Clear $Post first, if you have set it. There's no point to using both.
$PrePrint = If[TrueQ[ByteCount[#] > $OutputSizeLimit],
Iconize[#],
#] &;
This freezes the Front End for ~30 sec., but % can be used:
SeedRandom[0];
RandomInteger[1, 10^9] (* BEWARE: Freezes Front End ~30 sec *)
Total[%]
(* 500012762 *)
