With Custom REPL & Access to Kernel From Outside Process
Non-Blocking Version Update
There were issues with the front-end somehow (I really have no idea how) accessing the kernel outside of the preemptive link/service link or the standard packet channel (I assume) which caused things like usage-template lookup to hang the entire system.
To that effect I introduced a non-blocking version of this which frees us from having to manually control the kernel responses to the front-end, but also doesn't allow us to control/snoop on the front-end / kernel communication.
On the other hand we can still get STDIN control, so we can do things like define a notebook in the notebook interface and get it out of the process. Here's a picture of that in action:
This is now the default as it's much more robust than the blocking version. Until I figure out how to make the FE not try to access the kernel via a channel I don't control this will have to do.
Code
I added support for accessing the kernel state from a process outside Mathematica as well, using the OpenRead["!cat"] hack to get at stdin so I could also process that in my REPL.
It's still not super robust, but, again, it's a start. In particular I think if I can get all the packet handling to work as it should stop hanging. That'll take work, though, unless someone who has used LinkSnooper in the past has a record of what the response should be for every type of packet passed through.
I also changed the package name so that it's clear that it works with things other than IntelliJ (although via the $SubprocessKernelName variable you can give the kernel whatever name you want).
BeginPackage["SubprocessKernel`"]
$SubprocessFrontEnd::usage="The front end link for the kernel";
$SubprocessKernel::usage="The kernel link for the kernel";
$SubprocessKernelName::usage="The basic kernel name for the subprocess";
$SubprocessKernelExtension::usage="The extension for the kernel name";
OpenSubprocessNotebook::usage="Opens an a subprocess notebook";
StartSubprocessREPL::usage="Starts the subprocess REPL";
$SubprocessREPLSettings::usage="Settings for the subprocess REPL";
Begin["`Private`"]
startSubprocessFrontEnd[]:=
If[Quiet[$FrontEnd[[1]]=!=$SubprocessFrontEnd||!BooleanQ@LinkReadyQ[$SubprocessFrontEnd]],
Quiet@LinkClose[$SubprocessFrontEnd];
UsingFrontEnd;
launchFECommand=
System`UseFrontEndDump`Command["Server"->False]<>
Map[{" ",#1}&,
Join[System`UseFrontEndDump`Flags["Server"->False],{"-wstp"}]
];
$SubprocessFrontEnd=LinkLaunch[launchFECommand];
MathLink`SetFrontEnd[$SubprocessFrontEnd]
];
If[!StringQ@$SubprocessKernelName,
$SubprocessKernelName="Subprocess"];
If[!StringQ@$SubprocessKernelExtension,
$SubprocessKernelExtension="+"<>IntegerString[RandomInteger[50000], 16, 4]];
startSubprocessKernel[]:=
If[!MatchQ[$SubprocessKernel, _LinkObject?(Quiet[BooleanQ@LinkReadyQ[#]]&)],
$subprocessKernelName=
$SubprocessKernelName<>$SubprocessKernelExtension;
$SubprocessKernel=Quiet@LinkCreate[$subprocessKernelName, "LinkMode"->Listen],
$SubprocessKernel
];
configureSubprocessFrontEnd[]:=
(
$cachedKernelConfiguration=
CurrentValue[$FrontEndSession, {EvaluatorNames, $SubprocessKernelName}];
CurrentValue[$FrontEndSession, {EvaluatorNames, $SubprocessKernelName}]=
{
"AutoStartOnLaunch"->False,
"MLOpenArguments"->
TemplateApply[
"-LinkMode Connect -LinkProtocol SharedMemory -LinkName \"``\" -LinkOptions 256",
$subprocessKernelName
]
};
)
$SubprocessREPLSettings=
<|
"InitializationMessage"->"Starting kernel...",
"ProcessPacket"->processPacket,
"ProcessInput"->Function[{text}, processPacket@HoldComplete[EnterTextPacket[text]]],
"LinkWrite"->
Function[{link, response, packet},
(*Print@{packet, response};*)
LinkWrite[link, #]&/@response
],
"STDOUTWrite"->
Function[
{output, input},
Print[">> ------ Output: `` ------ >>"~TemplateApply~$Line];
Print[ToString[#, InputForm]<>"\n"]&/@output;
Print["<< ------ Output: `` ------ <<"~TemplateApply~$Line]
],
"RouteOutput"->
<|
"STDIN"->"STDOUT",
"FrontEnd"->"Link"
|>,
"LogPackets"->False,
"PollTime"->.1,
"Blocking"->False
|>;
evaluateThings[e_]:=
Reap[
GeneralUtilities`WithMessageHandler[
CheckAbort[ReleaseHold@e, $Aborted],
With[
{
sym=Extract[#, {2, "MessageTemplate", 1}, Hold],
name=Extract[#, {2, "MessageTemplate", 2}],
params=Sequence@@Extract[#, {2, "MessageParameters"}]
},
Sow[MessagePacket[sym, name]/.Hold[s_]:>s];
Sow[TextPacket@
Replace[
Internal`MessageButtonHandler[
MessageName[Evaluate@ReleaseHold[sym], name],
Hold[MessageName[sym, name]]/.Hold[s_Symbol]:>s,
Hold[Message[MessageName[sym, name], params]]/.Hold[s_Symbol]:>s
],
$Failed:>
ToString[HoldForm[MessageName[sym, name]]/.Hold[s_Symbol]:>s]
<>" "<>
ToString@
StringForm[
Replace[
Extract[#, {2, "MessageTemplate"}],
Except[_String]:>MessageName[General, name]
],
params
]
]
]
]&
]
]
standardEvaluationProcedure[expr_, makePacket_]:=
Module[{evRes=evaluateThings[expr], e, errs},
$incrementLine=True;
e=evRes[[1]];
errs=evRes[[2]];
{
errs,
Switch[e,
Null,
{},
_ErrorBox,
{
OutputNamePacket["Out[``]="~TemplateApply~$Line],
makePacket@e
},
_,
{
OutputNamePacket["Out[``]="~TemplateApply~$Line],
makePacket@e
}
]
}
]
logEvent[packet_]:=
With[
{
log=FileNameJoin@{$TemporaryDirectory, "subproc_repl_log.txt"}
},
If[!FileExistsQ@log, CreateFile@log];
OpenAppend[log];
WriteString[log,
StringRepeat["\n", 2]<>
StringRepeat[" ", 5]<>
StringRepeat["-", 25]<>
StringRepeat["\n", 2]
];
PutAppend[packet, log];
Quiet@Close@log;
];
logEcho[event_]:=
(logEvent[event];event)
processPacket//Clear
processPacket[p:HoldComplete@EvaluatePacket[res_]]:=
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEcho, Identity][
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEvent[p]];
ReturnPacket[res]
];
processPacket[p:HoldComplete@EnterExpressionPacket[expr_]]:=
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEcho, Identity][
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEvent[p]];
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEcho, Identity]@
standardEvaluationProcedure[
expr,
ReturnExpressionPacket[BoxData@If[Head@#===ErrorBox, #, ToBoxes@#]]&
]
];
processPacket[p:HoldComplete@EnterTextPacket[expr_]]:=
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEcho, Identity][
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEvent[p]];
standardEvaluationProcedure[
MakeExpression[expr, StandardForm],
If[Head@#===ErrorBox,
{},
ReturnTextPacket[ToString@#]
]&
]
];
processPacket[p:HoldComplete@InputPacket[expr_]]:=
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEcho, Identity][
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEvent[p]];
$incrementLine=True;
ReturnTextPacket[expr]
];
processPacket[p:HoldComplete[packet_]]:=
If[TrueQ@$SubprocessREPLSettings["LogPackets"], logEcho, Identity][
logEvent[p];
ReturnExpressionPacket[BoxData@ToBoxes@packet]
]
exitSubprocessREPL[]:=
(
$ParentLink=Null;
Quiet[LinkClose@$SubprocessKernel];
Quiet[LinkClose@$SubprocessFrontEnd];
)
packetWrite[response_, packet_, src_, link_, linkWrite_, stdoutWrite_]:=
If[response=!=None,
If[src==="STDOUT",
Block[{$ParentLink=Null},
stdoutWrite[
Flatten[{response}],
packet
]
];,
TimeConstrained[
linkWrite[
link,
Flatten[{response}],
packet
],
.1
]
]
]
subprocessBlockingREPL[]:=
Module[
{
kernelLink=$SubprocessKernel,
packet,
response,
stdin=OpenRead["!cat", BinaryFormat->True],
input,
output,
preemptiveLink=MathLink`$PreemptiveLink,
ppacket,
pevalStack=<||>,
pevalNum=1,
wroteToLink,
linkWrite:=$SubprocessREPLSettings["LinkWrite"],
processPacket:=$SubprocessREPLSettings["ProcessPacket"],
processInput:=$SubprocessREPLSettings["ProcessInput"],
stdoutWrite:=$SubprocessREPLSettings["STDOUTWrite"],
stdinOutput:=$SubprocessREPLSettings["RouteOutput", "STDIN"],
feOutput:=$SubprocessREPLSettings["RouteOutput", "FrontEnd"],
pollTime:=$SubprocessREPLSettings["PollTime"]
},
(*logEvent@"Entering loop";*)
While[True
(*Quiet@TrueQ[LinkConnectedQ@$SubprocessFrontEnd]&&
Quiet@TrueQ[LinkConnectedQ@$SubprocessKernel]*),
output=None;
response=None;
input=ReadString[stdin, EndOfBuffer];
(*logEvent["Link Ready: "<>ToString@LinkReadyQ@kernelLink];*)
If[Quiet@LinkReadyQ@preemptiveLink,
ppacket=TimeConstrained[LinkRead[preemptiveLink, HoldComplete], .5];
With[{n=pevalNum},
pevalStack[n]=ppacket;
If[$VersionNumber<11.2,
Function[Null, RunScheduledTask[#, {.0001}], HoldFirst],
SessionSubmit
][
pevalStack[n]={processPacket@pevalStack[n], pevalStack[n]};
While[KeyExistsQ[pevalStack, n-1],
Pause[.0001];
];
packetWrite[
pevalStack[n][[1]], pevalStack[n][[2]], "Link", preemptiveLink,
linkWrite, stdoutWrite
];
pevalStack[n]=.
]
]
];
If[StringQ@input&&StringLength@input>0,
output=processInput@input;
];
If[Quiet@LinkReadyQ@kernelLink,
(*logEvent["Link Read"];*)
packet=TimeConstrained[LinkRead[kernelLink, HoldComplete], .5];
response=processPacket[packet];
];
If[
response===ReturnPacket[Quit]||
output===ReturnPacket[Quit],
exitSubprocessREPL[];
Break[]
];
packetWrite[
output, input, stdinOutput, kernelLink,
Function[wroteToLink=True;linkWrite[##]], stdoutWrite
];
packetWrite[
response, packet, feOutput, kernelLink,
Function[wroteToLink=True;linkWrite[##]], stdoutWrite
];
If[TrueQ@$incrementLine, $Line++];
$incrementLine=False;
If[Head@response=!=ReturnPacket&&wroteToLink,
TimeConstrained[
LinkWrite[
$SubprocessKernel,
InputNamePacket["In[``]:="~TemplateApply~$Line]
],
.1
]
];
If[pollTime>0,
Pause[pollTime]
]
]
];
subprocessNonBlockingREPL[]:=
Module[
{
stdin=OpenRead["!cat", BinaryFormat->True],
input,
output,
wroteToLink,
linkWrite:=$SubprocessREPLSettings["LinkWrite"],
processPacket:=$SubprocessREPLSettings["ProcessPacket"],
processInput:=$SubprocessREPLSettings["ProcessInput"],
stdoutWrite:=$SubprocessREPLSettings["STDOUTWrite"],
stdinOutput:=$SubprocessREPLSettings["RouteOutput", "STDIN"],
feOutput:=$SubprocessREPLSettings["RouteOutput", "FrontEnd"],
pollTime:=$SubprocessREPLSettings["PollTime"]
},
If[$VersionNumber<11.2,
RunScheduledTask,
Function[Null, SessionSubmit[ScheduledTask[##]], HoldFirst]
][
input=ReadString[stdin, EndOfBuffer];
output=None;
If[StringQ@input&&StringLength@input>0,
output=$SubprocessREPLSettings["ProcessInput"]@input;
];
If[output===ReturnPacket[Quit],
exitSubprocessREPL[];
If[$VersionNumber<11.2,
RemoveScheduledTask@$ScheduledTask,
TaskRemove@$CurrentTask
]
];
packetWrite[
output, input, stdinOutput, $SubprocessKernel,
Function[wroteToLink=True;linkWrite[##]], stdoutWrite
];
If[TrueQ@$incrementLine, $Line++];
If[wroteToLink&&Head@output=!=ReturnPacket,
TimeConstrained[
LinkWrite[
$SubprocessKernel,
InputNamePacket["In[``]:="~TemplateApply~$Line]
],
.1
]
];
$incrementLine=False;,
pollTime
]
]
StartSubprocessREPL[]:=
With[
{
blocking=$SubprocessREPLSettings["Blocking"]=!=False
},
$Line=1;
startSubprocessFrontEnd[];
startSubprocessKernel[];
FrontEndExecute@FrontEnd`EvaluatorStart[$SubprocessKernelName];
If[blocking,
TimeConstrained[
LinkWrite[$SubprocessKernel, InputNamePacket["In[1]:="]],
.1
]
];
If[StringQ@$SubprocessREPLSettings["InitializationMessage"],
Print@$SubprocessREPLSettings["InitializationMessage"]
];
$ParentLink=$SubprocessKernel;
If[blocking,
subprocessBlockingREPL[];,
logEvent["REPL Started"];
subprocessNonBlockingREPL[]
];
]
OpenSubprocessNotebook[retry:True|False:True]:=
Module[{res},
startSubprocessFrontEnd[];
startSubprocessKernel[];
res=CreateDocument[{}, Evaluator->$SubprocessKernelName];
Which[
res===$Failed&&retry,
Quiet@LinkClose@$SubprocessKernel;
$SubprocessKernel=.;
Quiet@LinkClose@$SubprocessFrontEnd;
$SubprocessFrontEnd=.;
OpenSubprocessNotebook[],
res===$Failed,
$Failed,
True,
configureSubprocessFrontEnd[];
StartSubprocessREPL[];
res
]
]
End[]
EndPackage[]
This opens the notebook and we can almost use it like normal:
but there are a few key differences. For one, most packets that I don't explicitly handle cause it to hang indefinitely and the number of those is large. For another, we have access to exactly how the boxes are parsed in the REPL. Finally, hopefully someday we'll also be able to inject code from an external process into the kernel that this notebook is bound to. Once I figure out how to make Input work only for the stdin in the command prompt we should be golden.
No Custom Layer (Robust)
Here's a partial solution that's worth getting out here with notes for how it needs to be improved.
I just made a little package that attaches a front-end to a custom kernel. I assumed this was for the IntelliJ plugin so I named it that:
BeginPackage["IntelliJ`"]
$IntelliJFrontEnd::usage="The front end link for IntelliJ";
$IntelliJKernel::usage="The kernel link for IntelliJ";
OpenIntelliJNotebook::usage="Opens an IntelliJ Notebook";
StartIntelliJREPL::usage="Starts the IntelliJ REPL";
$IntelliJREPLSettings::usage="Settings for the IntelliJ REPL";
Begin["`Private`"]
startIntelliJFrontEnd[]:=
If[Quiet[$FrontEnd[[1]]=!=$IntelliJFrontEnd||!BooleanQ@LinkReadyQ[$IntelliJFrontEnd]],
Quiet@LinkClose[$IntelliJFrontEnd];
UsingFrontEnd;
launchFECommand=
System`UseFrontEndDump`Command["Server"->False]<>
Map[{" ",#1}&,
Join[System`UseFrontEndDump`Flags["Server"->False],{"-wstp"}]
];
$IntelliJFrontEnd=LinkLaunch[launchFECommand];
MathLink`SetFrontEnd[$IntelliJFrontEnd]
];
startIntelliJKernel[]:=
If[!MatchQ[$IntelliJKernel, _LinkObject?(Quiet[BooleanQ@LinkReadyQ[#]]&)],
$IntelliJKernel=Quiet@LinkCreate["IntelliJKernel", "LinkMode"->Listen],
$IntelliJKernel
];
configureIntelliJFrontEnd[]:=
(
CurrentValue[$FrontEnd, {EvaluatorNames, "IntelliJ"}]=
{
"AutoStartOnLaunch"->False,
"MLOpenArguments"->
"-LinkMode Connect -LinkProtocol SharedMemory -LinkName \"IntelliJKernel\" -LinkOptions 256"
};
)
$IntelliJREPLSettings=
<|
"PacketHandler"->
Function[{packet, kernel, fe},
Print@packet;
LinkWrite[fe, packet]
],
"PollTime"->.1
|>;
StartIntelliJREPL[]:=
Module[
{
replin,
replout,
startedEvaluator
},
startIntelliJFrontEnd[];
startIntelliJKernel[];(*
If[!TrueQ@startedEvaluator,
FrontEndExecute@FrontEnd`EvaluatorStart["IntelliJ"];
LinkWrite[$IntelliJKernel, InputNamePacket["IntelliJ init"]];
];*)
$ParentLink=$IntelliJKernel;
(*While[Quiet@TrueQ[LinkConnectedQ@$IntelliJFrontEnd],
If[LinkReadyQ[$IntelliJKernel, $IntelliJREPLSettings["PollTime"]],
replin=
Reap[
While[LinkReadyQ[$IntelliJKernel], Sow@Quiet@LinkRead[$IntelliJKernel]]
][[2]];
replout=
With[{evf=$IntelliJREPLSettings["PacketHandler"]},
If[#===$Failed,
Break[],
evf[
#,
$IntelliJKernel,
$IntelliJFrontEnd
]
]
]&/@replin
];
Pause[$IntelliJREPLSettings["PollTime"]]
]*)
]
OpenIntelliJNotebook[retry:True|False:True]:=
Module[{res},
startIntelliJFrontEnd[];
startIntelliJKernel[];
res=CreateDocument[{}, Evaluator->"IntelliJ"];
Which[
res===$Failed&&retry,
Quiet@LinkClose@$IntelliJKernel;
$IntelliJKernel=.;
Quiet@LinkClose@$IntelliJFrontEnd;
$IntelliJFrontEnd=.;
OpenIntelliJNotebook[],
res===$Failed,
$Failed,
True,
configureIntelliJFrontEnd[];
StartIntelliJREPL[];
res
]
]
End[]
EndPackage[]
The way I've been using it is loading it on in a terminal window:
Then this pops open a notebook with this kernel as its parent link and all the variables we defined in it accessible:
The trick was really to play with the EvaluatorNames via CurrentValue after creating the front-end. That way we can make a peer-to-peer connection and use that as the link the front-end starts itself with. Without that there was no way I could find in the MathLink API or the FrontEnd packet system to attach a LinkObject to a NotebookObject.
After that all I had to do was set the $ParentLink and it worked immediately.
Unfortunately this is only a partial solution as we've completely lost access to our original kernel from outside. We could possibly still access is by making another connection to the "IntelliJKernel" LinkObject but I'm not certain of this.
I also wanted to have a custom REPL working underneath where we could change the packet handler and also watch for stuff via Input (you can see I've got a bunch of commented out code there). For reasons I don't totally understand, though, this wasn't working. My inspiration for that was page 55 of this paper by David Wagner. He basically previewed a LinkSnooper like this there that I wanted to reimplement using a single kernel.
So this is a start, but there's work to be done.
Addendum
An interesting observation is that we can actually switch back and forth between terminal mode and notebook mode with this by setting $ParentLink back to Null or $IntelliJKernel:
We can of course see that the value of aaa I set in the notebook is kinda weird in the terminal, but it's an interesting, potentially very useful, thing to be able to do.
It also suggests that we could have multiple notebooks communicating with a single kernel in turn on different machines if the LinkObject were made to have a port and host parameter (as it can). Still, though, not quite what we want.
