17
$\begingroup$

I realized that there are lots of advantages to driving C/C++/FORTRAN code directly from Mathematica as LibraryLink functions (instead of running them from the command line or a shell script, as I have usually done before). This will give access to a lot of functionality that is difficult or time consuming to implement in a low level language (example).

There are disadvantages too, mainly because the running environment is typically a remote server, and not a local workstation (i.e. I don't get a notebook GUI). So, to get around some of the disadvantages,

Is the following feasible to implement (see below)?

Can we have a Mathematica master kernel running a remote kernel, running parallel calculations in subkernels, and do the following:

While the calculation is running, connect to the master computer through MathLink from a laptop; check the state of the calculations, perhaps do some quick-to-compute preliminary analysis on the so far calculated results; make decisions about continuing the calculation or not; then disconnect. It should be possible to connect to and disconnect from the server as many times as necessary without aborting the calculations for good.

Do you think that such a thing is (theoretically) possible to implement with the current version of Mathematica? Or perhaps the current features are not general enough to allow it?

Before sitting down to study the documentation in detail and try to implement this, I was wondering if anyone is aware of any showstopper limitations (or if anyone has tried to implement it).

Improve this question
$\endgroup$
1
  • $\begingroup$ It seems that I some time have the same situation if I understand your question correctly, I usually use Mathematica to drive fortran codes through librarylink. The workflow for me is something like this. I request several interactive nodes on the remote machine, and then launch a master kernel on one node, and connect back to my local frontend. The master kernel then manage the subkernels on the other nodes automatically. This way its much faster than use a VNC or X window forwarding especially if you have many plots in the notebook. $\endgroup$
    – xslittlegrass
    Commented Feb 3, 2015 at 1:38

5 Answers 5

Reset to default
9
$\begingroup$

Why MathLink or webMathematica? (both quite time-consuming once you do something nontrivial)

Keep it simple:

On Windows: Use Remote Desktop to connect to your server (where you started the FrontEnd, starting the parallel calculation).

On Linux: Use TightVNC or NX or some such.

Improve this answer
$\endgroup$
1
  • $\begingroup$ You are right ... I feel a little stupid. I can't use VNC on our server but screen could go a long way too. I could interrupt and go into a Dialog[] any time $\endgroup$
    – Szabolcs
    Commented Jan 26, 2012 at 23:28
8
$\begingroup$

Instead of driving a remote kernel directly via MathLink, you should consider implementing this task with webMathematica:

  • webMathematica 3 added support for queueing long running computations, which can also use compute kernels.
  • you can connect to webMathematica's web frontend with a web browser to inspect the state of the running computation at any time.
  • webMathematica already does all the handling of the kernel pool (launching and quitting kernels, re-launching crashed kernel, ...) which you would have to roll yourself.
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the reply! When you say that I can connect to webMathematica's web frontend to inspect the state, do you mean that I could check how much of the computation is done? Or could I look at arbitrary variables and do arbitrary (quick) computations on them? For just checking how far the calculation has progressed, there are other methods that are easy to implement. But what I describe would be useful for getting some information that I did not think of at the time when I started the computation. $\endgroup$
    – Szabolcs
    Commented Jan 24, 2012 at 20:14
  • $\begingroup$ +1. I thought about webMathematica, but I did not know about the features of version 3 that you mentioned. $\endgroup$
    – Leonid Shifrin
    Commented Jan 24, 2012 at 20:29
  • $\begingroup$ webMathematica lets you inspect the state (i.e., QUEUED, RUNNING, TERMINATED) of the computation. It's probably also possible to set session variables (see MSPSessionVariable) in you long running computation, which can run computations on in another request then. $\endgroup$
    – sakra
    Commented Jan 24, 2012 at 20:50
8
$\begingroup$

Here is a stab at a basic mathlink solution.

In the kernel that contains status information define this post function:

post[e_] := Module[{link},
 Quiet[
  link = LinkConnect["status"];
  If[LinkReadyQ[link], LinkRead[link]; LinkWrite[link, e]];
  LinkClose[link]
 ]]

If there is nothing to connect (there is no open "status" link on the system) then this will just (quietly) fail.

Now define a check function on the kernel that wants to know about the status of e on the other kernel:

check[] := Module[{link, res},
 link = LinkCreate["status"];
 LinkWrite[link, "hi there"];
 res = LinkRead[link];
 LinkClose[link];
 res]

When called, this function creates the status link and ping the other kernel with a LinkWrite (which blocks). When the other kernel connects and reads the "hi there" string it will send its value of e and the checking function receives it and closed the link and returns the result.

On my system I defined two extra kernels, like so:

SetOptions[$FrontEnd, EvaluatorNames -> {
 "Local" -> {"AutoStartOnLaunch" -> True}, 
 "K1" -> {"AutoStartOnLaunch" -> False}, 
 "K2" -> {"AutoStartOnLaunch" -> False}}]

(be careful, this may wipe out other kernels you may have defined previously).

Then I made two notebooks, one with a K1 kernel and one with a K2 kernel, evaluating the check[] function in the K1 kernel and the post[] function in the K2 kernel.

Then in the posting K2 kernel I added a scheduled task, as follows:

i = 1;
RunScheduledTask[post[i++]]

Which posts the status of i every second (and also increments it once per second).

Then every time I run check[] in the checking K1 kernel I get the current value of i.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is a very nice example! I'd like to suggest using $FrontEndSession in place of $FrontEnd to make the changes temporary. Question: Can we add a timeout to LinkRead in check? $\endgroup$
    – Szabolcs
    Commented Jan 27, 2012 at 11:33
  • $\begingroup$ And also to LinkWrite, is it possible to add a timeout? $\endgroup$
    – Szabolcs
    Commented Jan 27, 2012 at 13:00
  • 1
    $\begingroup$ I just found out that one could use CurrentValue to define the two new evaluators without wiping existing settings. Please see under Possible Issues in the SetOptions doc page. $\endgroup$
    – Szabolcs
    Commented Apr 21, 2012 at 20:07
5
$\begingroup$

From rather superficial looking at the docs, I think what you ask for should be possible and even relatively easy to implement, but I'd choose J/Link and Java over Mathlink, for the following reasons:

  • Java runtime is very stable. If you let it manage your remote kernels on the remote machine, you get a chance to have a better error / exception / crash handling. You can have a kernel pool as well, like WebMathematica does.

  • Java is more high-level than C, and J/Link more high-level than MathLink. This can be a big deal since JLink hides quite a bit of low-level MathLink complexity from you, and you might not need that low-level layer for your purposes. From what I saw in the docs (I only used J/Link to call Java code from Mathematica before, so have no experience with the other side yet), it does have all the necessary functionality.

I have worked with both MathLink and JLink (from the Mathematica calling other program side only), and with all my love for C (which, as a language, I prefer to Java), my experience has been that you can usually get things up and running in Java much quicker. I attribute this to the excellent design of J/Link, which hides lots of underlying complexities of MathLink.

You will probably need to know or learn some rudimentary Java and learn to work with some good IDE like Eclipse or IntelliJ Idea, if you choose the Java route. Eclipse is preferred, because WorkBench basically is Eclipse with Mathematica support, and you can have JLink projects where you can even debug Mathematica and Java code at the same time (which is another strong plus for Mathematica / Java combination). But, learning either one is not difficult.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I should really learn Java some day. I expect the language is not that difficult to learn, it might just take a lot of time to get familiar with its library. Did you ever open a MathLink connection, close it, then open it again? Anyway, this is not a project I am going to try to do now, but I had the impression it could be quite useful, if a sufficiently robust implementation is possible. $\endgroup$
    – Szabolcs
    Commented Jan 24, 2012 at 17:28
  • $\begingroup$ @Szabolcs I am lucky to have worked for a year as a professional Java programmer in the industry (on large banking software systems), but I learned Java initially on a few relatively simple projects. Pick one that interestes you and try to implement it all by yourself, then look at other available solutions. My all-time favorite is a virtual file system backed by a single file (I've written one which has some non-standard funky features and is pretty fast also - I actually thought of hooking it to Mathematica as an alternative to dbs in some cases). Java is a good one to know, if not to love. $\endgroup$
    – Leonid Shifrin
    Commented Jan 24, 2012 at 18:11
  • $\begingroup$ @Szabolcs As to your question on Mathlink - I did open and close connections many times when calling other programs from M, but that's probably not what you ask. $\endgroup$
    – Leonid Shifrin
    Commented Jan 24, 2012 at 18:12
  • $\begingroup$ @LeonidShifrin I can picture you implementing it all by yourself. Did you ever ask a question about MMA? Were you born answering your parent's doubts about life? Btw, your posts have already been very useful and interesting to me, greatly appreciated. The day you post a question you'll find 400 of us desperately trying to give something back (and probably failing). $\endgroup$
    – Rojo
    Commented Jan 24, 2012 at 18:58
  • $\begingroup$ I know nothing of Java and very little of C++/CLR .NET, and I'd love to learn some, mainly to extend the range of things I can do from MMA. So you think that in the present and near future it would be a better idea to learn Java, right? $\endgroup$
    – Rojo
    Commented Jan 24, 2012 at 19:00
1
$\begingroup$

I'm optimistic that it's theoretically possible, but it looks like it'll require custom code. It seems to me that a good starting point is using LinkCreate and LinkConnect. I'm not quite enough of an expert on the notebook interface to think of a way to trick it into remoting all of its input to the kernel on the other side of such a link, but it's not hard to imagine writing a simple console mode program that's capable of connecting to the session in progress.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I was thinking along the same lines, but I'm more worried about problems with parallelization. I will need to connect to the master kernel on the server while the subkernels are running. The master kernel should both periodically check for results coming back from subkernels (perhaps using the Parallel`Developer` functions), process them when it has time, and check for incoming connections from the outside, which should cause it to go into something like a Dialog. $\endgroup$
    – Szabolcs
    Commented Jan 24, 2012 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.