4
$\begingroup$

I'm trying to perform a computation that will:

  1. Be long on my machine (I estimate 24-48 hours)
  2. Consume a lot of memory
  3. Generate ~1 gig of data at the end

To manage the calculation I'd like to chunk it out, send it over mathlink to a slave kernel, do the computation, export the output, then quit and restart the slave kernel. This method has been discussed several times here and I've mostly relied on the posts here, here, and here.

A minimal example of my problem can be shown with a simplified version of newKernelEvaluate from this question.

ClearAll[newKernelEvaluate];
Attributes[newKernelEvaluate] = HoldAll;
newKernelEvaluate[expr_] := Module[{link, result},
   link = LinkLaunch[First@$CommandLine <> "  -subkernel"];
   LinkRead[link];
   LinkWrite[link, Unevaluated@EvaluatePacket@expr];
   result = LinkRead@link;
   LinkClose@link;
   Replace[result, ReturnPacket@x_ :> x]];

I'd like to wrap a Table in newKernelEvaluate and then wrap it with another Table. Like:

Table[ newKernelEvaluate[ Table[i, {i, j}]], {j, {{1, 2}, {3, 4}}}]

On my machine, the response I get is:

{MessagePacket[Table, "iterb"], MessagePacket[Table, "iterb"]}

But, I think I can get data into the slave kernel via this method because this works:

Table[newKernelEvaluate[k = j; k], {j, {{1, 2}, {3, 4}}}]

{{1, 2}, {3, 4}}


Not sure if this helps, but a bit more info about my specific problem. This is closer to what I've been working with:

Table[
 
 kernel = 
  LinkLaunch[First[$CommandLine] <> " -subkernel", 
   LinkProtocol -> "TCPIP"];
 LinkRead[kernel];
 LinkWrite[kernel, Unevaluated[qfrange = Range[qnum, qnum + 2]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, Unevaluated[nrange = Range[num, num + 2*5, 5]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, 
  Unevaluated[\[Sigma]range = 
    Sqrt[Range[\[Sigma]num, \[Sigma]num + 2*.05, .05]]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, 
  Unevaluated[
   Table[Module[{}, {qf, n, \[Sigma]}], {qf, qfrange}, {n, 
     nrange}, {\[Sigma], \[Sigma]range}]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, Unevaluated[Quit[]]];
 ,
 {qnum, {20, 23}},
 {num, {70, 85}},
 {\[Sigma]num, {1, 1.15}}
 ]

The table in this line:

LinkWrite[kernel, 
  Unevaluated[
   Table[Module[{}, {qf, n, \[Sigma]}], {qf, qfrange}, {n, 
     nrange}, {\[Sigma], \[Sigma]range}]]];

Contains a Module that will hold the calculation. As it happens, this bit of code works as expected on my machine, but when I add any functions that depend on the iterators in the inner Table it errors out. For example, this will send back error messages:

Table[
 
 kernel = 
  LinkLaunch[First[$CommandLine] <> " -subkernel", 
   LinkProtocol -> "TCPIP"];
 LinkRead[kernel];
 LinkWrite[kernel, Unevaluated[qfrange = Range[qnum, qnum + 2]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, Unevaluated[nrange = Range[num, num + 2*5, 5]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, 
  Unevaluated[\[Sigma]range = 
    Sqrt[Range[\[Sigma]num, \[Sigma]num + 2*.05, .05]]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, 
  Unevaluated[
   Table[Module[{a}, a = RandomReal[{-1, 1}, n]; a], {qf, 
     qfrange}, {n, nrange}, {\[Sigma], \[Sigma]range}]]];
 Print[LinkRead[kernel]];
 LinkWrite[kernel, Unevaluated[Quit[]]];
 ,
 {qnum, {20, 23}},
 {num, {70, 85}},
 {\[Sigma]num, {1, 1.15}}
 ]

Also, just incase it's relevant:

$Version

"13.1.0 for Mac OS X ARM (64-bit) (June 16, 2022)"

Thanks for any help.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.