How can I terminate a MathKernel that is running an external program after a timeout?

Question

I'm using ParrallelDo to parallel run another program (windows executable) in 8 subkernels. Sometimes a SubKernel freezes. What I see in the task-manager is that one or two of the MathKernels don't finish executing but build up page faults. The task of the external program has already finished by then. I then terminate the MathKernel with the task-manager by hand and ParallelDo redistributes the tasks - sometimes successful.

The external program is not written by me - so you can view it as a black box. I call it about 300-2000 times for one simulation which in total can take up to 8 hours - so I want to automatize this process. I want to terminate a subkernel after a specific time (i.e. four times the average time of the last program calls) in order to give ParallelTable a chance to redistribute the tasks.

Is this possible? And is this a good approach?

I tried TimeConstrained[] after I saw Pillsys answer but it didn't work for me.

Here are the relevant code snippets:

ParallelDo[
    ...
    PutAppend[...first comment...];
    TimeConstained[RunB["program.exe"<>item],600];
    PutAppend[...second comment...];
    ...
,{item,items}]

with

RunB[cmd_String]:= Module[{shell},
    shell=NETLink'CreateCOMObject["WScript.shell"];
    shell@run[cmd,0,True]
];

After a while I see this in the taskmanager ("Seitenfehler" means page faults):

but the program.exe does not appear in the list so it should have finished. However, Mathematica doesn't reach the second comment so it must still be within the TimeConstrained[] command.

Any thoughts/ideas to how I can kill the task?

Answer 1

Without the program in question, I had to fake it using sleep (which may not even exist on Windows), but here's a rough outline that I believe will work for your needs. I've been working on this sort of problem a lot recently, for similar reasons, so this is all going to be a little on the elaborate side.

Here's a shell command that will invoke it on Unix-y systems, but I hope the idea translates:

ClearAll[sleepy, timeQ, toSeconds, timeConstrainedProcess];

timeQ[q_Quantity] := UnitDimensions[q] === {{"TimeUnit", 1}};
toSeconds[q_Quantity?timeQ] := QuantityMagnitude[q, "Seconds"];

sleepy[q_Quantity?timeQ] := sleepy[toSeconds[q]];
sleepy[seconds_?NonNegative] :=
  StringTemplate["sleep `` && echo 'awake now!'"][seconds];   

Then it looks like TimeConstrained will indeed work, and at least on the Mac, the offending process will be killed.

TimeConstrained[
 RunProcess[{$SystemShell, "-c", sleepy[Quantity[1, "Seconds"]]}], 10]
(* <|"ExitCode" -> 0, "StandardOutput" -> "awake now!\n", 
     "StandardError" -> ""|> *)

TimeConstrained[
 RunProcess[{$SystemShell, "-c", sleepy[Quantity[1, "Minutes"]]}], 10]
(* $Aborted *)

To check your use case, I went ahead and tested it out with parallel kernels; however, I had the TimeConstrained be part of the expression passed to the subkernels, which worked like a charm.

LaunchKernels[];
DistributeDefinitions[sleepy, timeQ];

I don't know for sure why I had to distribute timeQ to the subkernels, but it wasn't automatically distributed. I'll have to search the SE archives later. Now for the moment of truth:

ParallelMap[
  TimeConstrained[
    StringTrim@
     RunProcess[{$SystemShell, "-c", sleepy[#]}]["StandardOutput"], 
    10] &,
  {1, 5, 15, 25}] // AbsoluteTiming
(* {10.0157, {"awake now!", "awake now!", $Aborted, $Aborted}} *)

---

UPDATE: I did a little more experimenting, and one thing I found is that Run does not work properly with TimeConstrained, unlike RunProcess. This probably falls short of being a bug, but it's surprising and dubious, IMO.

TimeConstrained[Run["sleep 10"], 5] // AbsoluteTiming
(* {10.0079, 0} *)

TimeConstrained[RunProcess[{"sleep", "10"}], 5] // AbsoluteTiming
(* {5.00194, $Aborted} *)

Replacing Run with RunProcess may resolve your issue.

Answer 2

cmd script

I tried Pillsys Method but didn't have any success. So I tried a different route by writing a batch file that will monitor the MathKernels and kills them if they have a page fault of over one million. Credit goes to DavidPostill for pointing me to Sysinternals (see here).

1. Download Sysinternals Suite from https://technet.microsoft.com/de-de/sysinternals/bb842062.aspx
2. Make the programs visible by adding the folder (in my case ";d:\scripts\SysintervalsSuite\") to the PATH variable in Windows

3. Add the folowing script in "d:\scripts\checkmathkernels.bat":

   @echo off
   SETLOCAL ENABLEEXTENSIONS
   setlocal EnableDelayedExpansion
   REM Dieses script soll MathKernels schließen die eine zu hohe (>1M) page fault haben. Es prüft alle 60 Sekunden.
   REM Der folgende Aufruf liest alle MathKernel Prozesse aus. PIDs sind an Stelle 2. (Page)Faults sind an Stelle 7.
   REM pslist -m -nobanner MathKernel | findstr /C:"MathKernel"
   :start
   set "output_cnt=0"
   for /f "tokens=2,7 " %%e in ('pslist -m -nobanner MathKernel ^| findstr /C:"MathKernel"') do (
       set /a output_cnt+=1    
       set "pids[!output_cnt!]=%%e"    
       set "faults[!output_cnt!]=%%f"    
   )
   REM Ausgabe:
   echo PID  faults
   for /L %%n in (1 1 !output_cnt!) DO (
       if !faults[%%n]! GTR 1000000 (
       call taskkill /f /PID !pids[%%n]!
   )
   echo !pids[%%n]! !faults[%%n]!
   )
   timeout /T 60
   goto start
   

4. Call bat from Mathematica or start it with a doubleclick

PowerShell

If you want to avoid using Sysintervals you could use the following PowerShell Script (tested on Win7 SP1, PowerShell v5):

$counter = 0;    
Do {
$pr = Get-WmiObject -Query "select * from win32_process where name='MathKernel.exe'"    
foreach ($i in $pr) {
    Write-Output "Process $($i.Handle) has $($i.PageFaults) PageFaults"
    If ($i.PageFaults -ge 1000000) {        
        Stop-Process $i.Handle -Force
        $counter++
        Write-Output "Process $($i.Handle) stopped"
    }
}
Write-Output "$($counter) processes stopped so far"
Start-Sleep -Seconds 60
} Until ( $Host.UI.RawUI.KeyAvailable -and ($Host.UI.RawUI.ReadKey("IncludeKeyUp,NoEcho").VirtualKeyCode -eq 27 ) )  

If you save it as "pf1.sh1" you can call it from a bat file in the same directory with

powershell.exe -noprofile -executionpolicy bypass -file %~dp0\pf1.ps1