6
$\begingroup$

I'm using ParrallelDo to parallel run another program (windows executable) in 8 subkernels. Sometimes a SubKernel freezes. What I see in the task-manager is that one or two of the MathKernels don't finish executing but build up page faults. The task of the external program has already finished by then. I then terminate the MathKernel with the task-manager by hand and ParallelDo redistributes the tasks - sometimes successful.

The external program is not written by me - so you can view it as a black box. I call it about 300-2000 times for one simulation which in total can take up to 8 hours - so I want to automatize this process. I want to terminate a subkernel after a specific time (i.e. four times the average time of the last program calls) in order to give ParallelTable a chance to redistribute the tasks.

Is this possible? And is this a good approach?

I tried TimeConstrained[] after I saw Pillsys answer but it didn't work for me.

Here are the relevant code snippets:

ParallelDo[
    ...
    PutAppend[...first comment...];
    TimeConstained[RunB["program.exe"<>item],600];
    PutAppend[...second comment...];
    ...
,{item,items}]

with

RunB[cmd_String]:= Module[{shell},
    shell=NETLink'CreateCOMObject["WScript.shell"];
    shell@run[cmd,0,True]
];

After a while I see this in the taskmanager ("Seitenfehler" means page faults): TaskManager with stucked MathKernel

but the program.exe does not appear in the list so it should have finished. However, Mathematica doesn't reach the second comment so it must still be within the TimeConstrained[] command.

Any thoughts/ideas to how I can kill the task?

$\endgroup$
  • $\begingroup$ Two questions: 1. Is RunB a typo? 2. Have you tried using RunProcess? $\endgroup$ – Pillsy May 3 '17 at 14:51
  • $\begingroup$ @Pillsy No RunB is a Method I defined as described above. I cut the part out were it switches to Run[] for other $OperatingSystem than Windows. It basically means "Run in Background". $\endgroup$ – Kab May 5 '17 at 15:14
  • $\begingroup$ Note that ReadString has its own TimeConstraint option. $\endgroup$ – Alexey Popkov May 9 '17 at 4:47
  • $\begingroup$ Strongly related: "How to kill slave kernel securely?" $\endgroup$ – Alexey Popkov Jul 18 '17 at 15:05
  • $\begingroup$ I don't use ReadString in my script. "How to kill slave kernel securely?" describes how to kill a slave kernel. However, I have to know when this is necessary - hence the timeout approach. Can I monitor the MathKernels paging error pool within Mathematica and kill them when necessary? $\endgroup$ – Kab Jul 20 '17 at 10:06
3
$\begingroup$

Without the program in question, I had to fake it using sleep (which may not even exist on Windows), but here's a rough outline that I believe will work for your needs. I've been working on this sort of problem a lot recently, for similar reasons, so this is all going to be a little on the elaborate side.

Here's a shell command that will invoke it on Unix-y systems, but I hope the idea translates:

ClearAll[sleepy, timeQ, toSeconds, timeConstrainedProcess];

timeQ[q_Quantity] := UnitDimensions[q] === {{"TimeUnit", 1}};
toSeconds[q_Quantity?timeQ] := QuantityMagnitude[q, "Seconds"];

sleepy[q_Quantity?timeQ] := sleepy[toSeconds[q]];
sleepy[seconds_?NonNegative] :=
  StringTemplate["sleep `` && echo 'awake now!'"][seconds];   

Then it looks like TimeConstrained will indeed work, and at least on the Mac, the offending process will be killed.

TimeConstrained[
 RunProcess[{$SystemShell, "-c", sleepy[Quantity[1, "Seconds"]]}], 10]
(* <|"ExitCode" -> 0, "StandardOutput" -> "awake now!\n", 
     "StandardError" -> ""|> *)

TimeConstrained[
 RunProcess[{$SystemShell, "-c", sleepy[Quantity[1, "Minutes"]]}], 10]
(* $Aborted *)

To check your use case, I went ahead and tested it out with parallel kernels; however, I had the TimeConstrained be part of the expression passed to the subkernels, which worked like a charm.

LaunchKernels[];
DistributeDefinitions[sleepy, timeQ];

I don't know for sure why I had to distribute timeQ to the subkernels, but it wasn't automatically distributed. I'll have to search the SE archives later. Now for the moment of truth:

ParallelMap[
  TimeConstrained[
    StringTrim@
     RunProcess[{$SystemShell, "-c", sleepy[#]}]["StandardOutput"], 
    10] &,
  {1, 5, 15, 25}] // AbsoluteTiming
(* {10.0157, {"awake now!", "awake now!", $Aborted, $Aborted}} *)

UPDATE: I did a little more experimenting, and one thing I found is that Run does not work properly with TimeConstrained, unlike RunProcess. This probably falls short of being a bug, but it's surprising and dubious, IMO.

TimeConstrained[Run["sleep 10"], 5] // AbsoluteTiming
(* {10.0079, 0} *)

TimeConstrained[RunProcess[{"sleep", "10"}], 5] // AbsoluteTiming
(* {5.00194, $Aborted} *)

Replacing Run with RunProcess may resolve your issue.

$\endgroup$
  • $\begingroup$ Thanks for your comment. I had to wait for the problem to occur. I wrapped my program call in TimeConstrained but with no effect. Within the ParrallelDo code I write a comment in a file, have the TimeContrained[Run...] line and write another comment. It's now stuck for way longer than the time limit, the external program task is finished but second comment hasn't been written. To hide the run I used: shell=NETLink'CreateCOMObject["Wscript.shell"]; shell@run[cmd,0,True] with cmd being a string containing the program name with parameters. Any ideas? $\endgroup$ – Kab May 3 '17 at 9:05
  • $\begingroup$ Another Question: I don't use 'DistributeDefinitions[]' at all. And it seems not to be neccessary. Most of the time my scripts works like a charm. I'm using Windows7 with Mathematica 10.0 so I can't directly replicate your results with your code. What happens when you omit DistributeDefinitions[] ? $\endgroup$ – Kab May 3 '17 at 9:48
  • $\begingroup$ Nothing in particular; I just needed it to get the definition of sleepy to the subkernels. $\endgroup$ – Pillsy May 3 '17 at 14:50
0
$\begingroup$

cmd script

I tried Pillsys Method but didn't have any success. So I tried a different route by writing a batch file that will monitor the MathKernels and kills them if they have a page fault of over one million. Credit goes to DavidPostill for pointing me to Sysinternals (see here).

  1. Download Sysinternals Suite from https://technet.microsoft.com/de-de/sysinternals/bb842062.aspx
  2. Make the programs visible by adding the folder (in my case ";d:\scripts\SysintervalsSuite\") to the PATH variable in Windows
  3. Add the folowing script in "d:\scripts\checkmathkernels.bat":

    @echo off
    SETLOCAL ENABLEEXTENSIONS
    setlocal EnableDelayedExpansion
    REM Dieses script soll MathKernels schließen die eine zu hohe (>1M) page fault haben. Es prüft alle 60 Sekunden.
    REM Der folgende Aufruf liest alle MathKernel Prozesse aus. PIDs sind an Stelle 2. (Page)Faults sind an Stelle 7.
    REM pslist -m -nobanner MathKernel | findstr /C:"MathKernel"
    :start
    set "output_cnt=0"
    for /f "tokens=2,7 " %%e in ('pslist -m -nobanner MathKernel ^| findstr /C:"MathKernel"') do (
        set /a output_cnt+=1    
        set "pids[!output_cnt!]=%%e"    
        set "faults[!output_cnt!]=%%f"    
    )
    REM Ausgabe:
    echo PID  faults
    for /L %%n in (1 1 !output_cnt!) DO (
        if !faults[%%n]! GTR 1000000 (
        call taskkill /f /PID !pids[%%n]!
    )
    echo !pids[%%n]! !faults[%%n]!
    )
    timeout /T 60
    goto start
    
  4. Call bat from Mathematica or start it with a doubleclick

PowerShell

If you want to avoid using Sysintervals you could use the following PowerShell Script (tested on Win7 SP1, PowerShell v5):

$counter = 0;    
Do {
$pr = Get-WmiObject -Query "select * from win32_process where name='MathKernel.exe'"    
foreach ($i in $pr) {
    Write-Output "Process $($i.Handle) has $($i.PageFaults) PageFaults"
    If ($i.PageFaults -ge 1000000) {        
        Stop-Process $i.Handle -Force
        $counter++
        Write-Output "Process $($i.Handle) stopped"
    }
}
Write-Output "$($counter) processes stopped so far"
Start-Sleep -Seconds 60
} Until ( $Host.UI.RawUI.KeyAvailable -and ($Host.UI.RawUI.ReadKey("IncludeKeyUp,NoEcho").VirtualKeyCode -eq 27 ) )  

If you save it as "pf1.sh1" you can call it from a bat file in the same directory with

powershell.exe -noprofile -executionpolicy bypass -file %~dp0\pf1.ps1
$\endgroup$
  • $\begingroup$ You can try krnKill from this answer. $\endgroup$ – Alexey Popkov Jul 18 '17 at 21:49
  • $\begingroup$ Hi Alexey Popkov. I understand your question in this link as related to slave kernels on other machienes. Is this right? Would I need this with local kernels? $\endgroup$ – Kab Jul 20 '17 at 9:57
  • $\begingroup$ No, my question was primarily about slave kernels on the master machine and my own solutions were all tested only on master kernel's machine. $\endgroup$ – Alexey Popkov Jul 20 '17 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.