I'm using webMathematica, my license supports a maximum of two kernels running at the same time. Sometime an error occurs, I've seen three at least (from webMathematica.log):
- [ERROR] MathLink connection was lost
- [ERROR] Connected MathLink program has closed the link //see http://mathematica.stackexchange.com/questions/104053/license-not-released-after-kernel-is-stopped-for-kerneltimelimit-reached
- Another one I can't remember (still a MathLinkException)...
... And whatever causes them, then the system seems to be totally unable to recover and starts spamming the error:[INFO ] Setting HTTP error 403 to all subsequent calls. I suppose it does so because of the dying kernel not releasing the license (I'm not really sure).
I'm not asking how to prevent the crashes, what I'm asking is how can I handle them so that the system can recover and just few requests are "broken".
The exception is not thrown by my code, it's webMathematica that manages its kernels, I don't really know where to start...
EDIT Wolfram response:
[...] The issue you are seeing arises because of crashed Mathematica kernels that do not release the license. Hence all the the following webmathematica requests are refused because the Kernel manager has no more licenses for available kernels. Although the housekeeping code should clean up the crashed kernels, unfortunately that is not happening for some reason in certain setups and the only option remaining is to kill the hung kernels and restart Tomcat webserver manually (which you are doing already). [...]
As a temporary work-around I will implement something like: "If error is 403 restart tomcat" and a basic "Retry N times before giving up".
