21
$\begingroup$

I am running Mathematica on a remote Linux cluster, by the terminal, using the math command. (I do not have access to a terminal GUI. For example, when I type the command mathematica, I get this error message: Can't open display "".)

I write Mathematica code in a text file (e.g., test.txt) and pass it to math using either the command 

math < test.txt

or the command

math -script test.txt

Both of these methods work fine for starting a Mathematica job. However, when I log out of the terminal -- or even when a brief loss of internet connection interrupts my connection to the remote cluster -- the Mathematica script is terminated. When I log back in the terminal, no Mathematica processes (i.e., MathKernel) are running.

To prevent this termination upon logout, I have tried using the standard Linux program nohup, which is a command telling the terminal to ignore the HUP (hangup) signal; output is redirected to a file nohup.out. I use nohup with any of the following commands:

nohup math < test.txt

or

nohup math < test.txt &

or

nohup math -script test.txt

or

nohup math -script test.txt &

These commands all start the Mathematica job fine, but when I logout and log back in to the terminal, no MathKernel or similar processes are running (as verified by ps and by ps -u myusername). The situation is the same regardless of whether my job is a single kernel job or a parallel, multikernel job. Since the jobs start fine -- including those ending with the & symbol to force the job to the background -- I do not think that I am having the same "background job suspension" problem as in this question.

Do you have any thoughts on how I can prevent the terminal from terminating jobs when I logout or my connection is briefly interrupted? The nohup method above works for me with other programs, but for some reason not with Mathematica.

(A toy test example is Table[Length[FactorInteger[10^50 + n]], {n, 50}], which takes about 30 seconds on the terminal.)

|improve this question
$\endgroup$
7
$\begingroup$

Another option would be to use the at command to create detached jobs:

echo "math -script test.txt" | at now
|improve this answer
$\endgroup$
  • $\begingroup$ Very useful and short! $\endgroup$ – Chen Stats Yu Jan 31 '15 at 0:47
19
$\begingroup$

You need to daemonize your script:

nohup math -script test.txt 0<&- &>/dev/null &

Now this will run as a background process with no output captured. If your script does indeed produce output, just replace /dev/null with the filename.

In order to daemonize something you need to disconnect all the automatically connected streams (stdin, stdout, stderr).

I don't know really if this is a Mathematica question, since it is more unix/linux related, but I got tired of only giving comments. ;)

Another trick is just to double background your script, which will make nohup of no need:

(math -script test.txt &)& (* very /bin/bash related *)
|improve this answer
$\endgroup$
  • 3
    $\begingroup$ I hope the poor script won't feel too hurt by such slander :-) $\endgroup$ – Szabolcs Feb 24 '14 at 19:01
  • $\begingroup$ :) but since the daemon (where it all relates from) looks rather friendly upload.wikimedia.org/wikipedia/en/5/55/Bsd_daemon.jpg I would not fear too much on that script. $\endgroup$ – Stefan Feb 24 '14 at 19:31
  • $\begingroup$ @Stefan @Szabolcs Thanks so much to both of you. I know that I am getting more into a Linux question, but I don't completely understand why just nohup math -script & does not work. For some other programs that I use, this works and dumps all the output to a file called nohup.out. I would like to capture the output, so I will go with your first suggestion. If I want to capture to a file nohup.out, do I just do nohup math -script test.txt 0<&- &>./nohup.out &? $\endgroup$ – Andrew Feb 24 '14 at 22:37
  • 1
    $\begingroup$ @Andrew nohup is just catching the SIGHUP signal, but does not disconnect the standard file descriptors. depends on the software how much they rely on those. but to disconnect a session from a software which does hold a connection on them you must detach with all the file descriptors available. therefore you need that "ugly" trick. $\endgroup$ – Stefan Feb 24 '14 at 22:54
  • 1
    $\begingroup$ I'd rather do math -script test.txt & disown, where & puts the script into background, and disown detaches it from the current terminal - instead of all that manual magic with redirections and nohup, which would also create a nohup.out file. $\endgroup$ – Ruslan Feb 25 '14 at 10:02
14
$\begingroup$

You can use GNU screen to make a sort of persistent terminal that allows you to resume work wherever you left off. Take a look at the many tutorials available.

It's not completely clear from your question whether the better solution is this, or nohup (see Stefan's answer).

Use nohup if your workflow is non-interactive: log in, start a batch job that writes results to a file, log out, log back in the next day to retrieve results.

Use screen for an interactive workflow: log in, start math in interactive mode, disconnect because your internet connections is interrupted (or you need to go for lunch), return later to continue the interactive session.

Short tutorial on screen:

  1. Log in, and start screen.
  2. Start math, do interactive work.
  3. When you need to log out, first press Ctrl-A, D to detach from screen (you do not need to put math to the background). Then log out using exit. Do not log out using exit before detaching from screen, otherwise this will close screen for good.
  4. Log back in and start screen as screen -r. This returns you to where you left off.

I have in fact used this for running long jobs with Mathematica. The reason why I preferred to run math interactively instead of using it in batch mode to run scripts was that I could interrupt the evaluation temporarily using Ctrl-C, I to enter an interactive Dialog and examine the state of the calculation. (Note that that is a potentially dangerous thing to do, and how to do it safely would be the topic of a different Q/A... Just warning you.)

|improve this answer
$\endgroup$
  • $\begingroup$ Well, this is a really nice approach on an interactive workflow...especially the Dialog part, though dangerous, but it is indeed a nice option. Very elaborated indeed. $\endgroup$ – Stefan Feb 24 '14 at 18:14
  • $\begingroup$ @Stefan If I remember correctly, interrupting parallel calculations was not safe. I don't think I ran into problems otherwise, but be sure to test interruptibility on a short job before interrupting a job that would run for a week ... $\endgroup$ – Szabolcs Feb 24 '14 at 18:25
  • $\begingroup$ I always preferred screen to nohup as I could use multiple "windows" (screens?) to monitor progress, plus logging. :) $\endgroup$ – rcollyer Jan 21 '15 at 14:55
3
$\begingroup$

This is a shell dependent issue. With csh/tsch the & automatically nohup's, so just this works:

 math -script test.m < /dev/null &

There seems to be a mathematica/csh issue that the script hangs if disconnected form stdin even though it doesn't try to read anything

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks. I am using bash, so I don't think the above works for me, but I am sure it works for csh/tsch. $\endgroup$ – Andrew Feb 24 '14 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.