2
$\begingroup$

Using webMathematica, doing massive tests of our functions (each call can take from 0 to 15 seconds). After a while we started getting lots of 403 Forbidden errors.

webMathematica.log:

2016-01-13 12:25:12,510 com.wolfram.kerneltools.KernelPool <General kernelThread-1 > [INFO ] Kernel 263 stopped
2016-01-13 12:25:12,510 com.wolfram.kerneltools.KernelPool <General kernelThread-1 > [INFO ] Kernel 265 started
2016-01-13 12:25:57,377 com.wolfram.kerneltools.state.StateMachine <General Timer-267 264> [ERROR] Evaluation did not complete within the allowed time limit
2016-01-13 12:25:57,377 com.wolfram.msp.MSPKernelRequest <General http-bio-8082-exec-1 264> [INFO ] Setting HTTP error 403
2016-01-13 12:25:57,878 com.wolfram.kerneltools.KernelPool <General kernelThread-0 > [INFO ] Kernel 264 stopped
2016-01-13 12:25:57,878 com.wolfram.kerneltools.KernelPool <General kernelThread-0 > [INFO ] Kernel 266 started
2016-01-13 12:25:58,291 com.wolfram.kerneltools.state.StateMachine <General kernelThread-0 266> [ERROR] Connected MathLink program has closed the link, but there might still be data underway.
MathLinkException: 11: Connected MathLink program has closed the link, but there might still be data underway.
        at com.wolfram.jlink.NativeLink.nextPacket(NativeLink.java:289)
        at com.wolfram.jlink.WrappedKernelLink.nextPacket(WrappedKernelLink.java:236)
        at com.wolfram.jlink.KernelLinkImpl.waitForAnswer(KernelLinkImpl.java:174)
        at com.wolfram.jlink.KernelLinkImpl.discardAnswer(KernelLinkImpl.java:191)
        at com.wolfram.kerneltools.Kernel.initialize1(Kernel.java:304)
        at com.wolfram.kerneltools.Kernel.initialize(Kernel.java:249)
        at com.wolfram.kerneltools.KernelPool$KernelRunnable.run(KernelPool.java:938)
        at java.lang.Thread.run(Thread.java:745)
2016-01-13 12:25:58,793 com.wolfram.kerneltools.KernelPool <General kernelThread-0 > [INFO ] Kernel 266 stopped
2016-01-13 12:26:02,879 com.wolfram.msp.MSPKernelRequest <General http-bio-8082-exec-1 > [INFO ] Setting HTTP error 403
2016-01-13 12:26:03,200 com.wolfram.msp.MSPKernelRequest <General http-bio-8082-exec-2 > [INFO ] Setting HTTP error 403

Extracting meaninful parts:

1) [ERROR] (kernel 264) Evaluation did not complete within the allowed time limit
2) [INFO ] Kernel 264 stopped
3) [INFO ] Kernel 266 started (spawned to take the place of 264)
4) [ERROR] (kernel 266) Connected MathLink program has closed the link [...]
5) [INFO ] Kernel 266 stopped
6) [INFO ] Setting HTTP error 403
7) [INFO ] Setting HTTP error 403
8) ... 403 error for five hours

The kernel that should have taken the place of the one that timed-out died immediatelly (license not realeased by 264?), from that point, most of the requests crashed with 403 error. Even after another kernel succesfully started.

I have functions that might excede the KernelTimeLimit, it's normal and I don't want to increase the limit. I just expected the single evaluation to fail, not to crash everything else for several hours. The situation eventually normalized without any human intervenction after about 5 hours.

Anyone know how can I prevent this behaviour?

$\endgroup$