# How to constrain memory usage on HPC?

I found the performance of built-in function MemoryConstrained is very poor. It stops the evaluation after the memory limit has already been exceeded greatly than the values I set.

There are several other questions on how to constrain memory usage during an evaluation. For example, this one. But the method seems not working for parallel situation(for example, if you performParallelTable), because MemoryInUse is specific to each kernel. No bother, there is another way, that is use Jlink to get the system's free physical memory information( see here ). So I combined these two method. Reach the following code:

Needs["JLink"];
InstallJava[];
intervalBetweenTests = 1;
JavaBlock[
javalangmanagementManagementFactorygetOperatingSystemMXBean[]\
@#[] &@getFreePhysicalMemorySize]) < 1*1024^3,
Quit[]], intervalBetweenTests]


After running this code, mathematica launch a watchdog, and any evaluation (no matter parallel or not) will be aborted if the free physical memory in current machine is below 1GB.

The code just mentioned works on a single machine. But I have to work on HPC, every parallel evaluation is performed on several different machine simultaneously, and I have to make sure that free physical memory won't below a threshold on every machine I use during a parallel evaluation, otherwise the machine will be dead, the admin will be very unhappy and mad about me:)

I tried ParallelEvaluate the above code, intended to launch watchdogs on all available kernels. That is

ParallelEvaluate[Needs["JLink"];
InstallJava[];
intervalBetweenTests = 1;
JavaBlock[
javalangmanagementManagementFactorygetOperatingSystemMXBean[]\
@#[] &@getFreePhysicalMemorySize]) < 1*1024^3,
Quit[]], intervalBetweenTests]]


but didn't work. The above code output a list of same ScheduledTask object, and evaluate ScheduledTasks[] shows no running scheduledtask at all. And of course the The free physical memory is not constrained as expected when doing calculation.

questions:

1. it seems that RunScheduledTask can't been parallelized, is that true?
2. why built-in function MemoryConstrained is so poor?
3. finally, how to achieve memory control across machines in mathemaica?

edit

as OleksandrR said, an HPC usually equipped with a job management system, it provide resource control of the job. for example in PBS job system, pmem can constrain the memory usage per process of a job. But to constrain memory usage inside mathematica itself is much more adaptive. There are cases

1. some small cluster doesn't equipped with a job management system
2. the way to use the HPC is variable. Very often we can use mathemaica interactively on HPC by submitting an interactive PBS job, and if the pmem is set at the beginning, then all the calculations doing in this interactive job is restricted per process, while we should have been able to use more memory if we didn't set the pmem when submit the job.

and other cases you can think of, but I think these two is enough.

-
Can you specify what you mean by "it didn't work"? Did you get an error message or did the approach simply not work as expected? –  halirutan Dec 7 '13 at 1:54
@halirutan No error message. It simply seems that the watchdog didn't launch, and free physical memory is not constrained, even not on HPC. –  matheorem Dec 7 '13 at 4:50
This is not much help for the question, but IMO, the administrator requiring users to monitor their own memory usage is a sign that they are not doing their job properly. It is the function of the cluster resource management system (e.g. Torque) to ensure that resources are properly used and that sensible resource limits are not exceeded. If this was set up properly, the problem would not arise in the first place. –  Oleksandr R. Dec 7 '13 at 18:21
@OleksandrR. Thank you so much! You're right.Generally, an HPC has job management system. For example, the PBSpro provide pmem option to constrain memory per process. I actually forgot this before, thank you for reminding me this. –  matheorem Dec 8 '13 at 3:07
I agree, it's better to do it through Mathematica if possible. If nothing else, with pmem, if your job exceeds the quota then it will just be killed immediately, which is not necessarily very helpful. But, proper configuration/usage of the resource manager should at least prevent one user's mistake impacting anyone else, so your administrator doesn't really have much cause for complaint if this happens, as I see it. –  Oleksandr R. Dec 8 '13 at 3:12

My tests on MacOSX with Mathematica 9.0.1 brought up the following answers to your questions.

1. it seems that RunScheduledTask can't been parallelized, is that true?

No, I think it works. First, I thought too that this might be the problem, but a quick test showed that it works as expected. For the following, please be careful to close the parallel kernels or quit between the examples.

LaunchKernels[2];

ParallelEvaluate[
i = 0;
];

Pause[5];
ParallelEvaluate[i]

(* {4, 4} *)


As you can see i is not defined on you main kernel and it is increased every second on the subkernels. Just call ParallelEvaluate[i] several times.

1. why built-in function MemoryConstrained is so poor?

I'm not sure how to answer this objectively. The only thing I can say is, that I almost never use memory constraining myself. I guess most of the users don't use this and this might be an indicator why it is poor.

1. finally, how to achieve memory control across machines in Mathematica?

You will use the approach you suggested where I will try to fix some things. My starting point was to first verify that RunScheduledTask is working properly on the sub-kernels. After this, the bug was very likely connected to JLink.

Therefore, I inspected your code more closes. If we simplify you second example just to see whether or not the java class is loaded at all, we see that

ParallelEvaluate[
InstallJava[];
Hold[cl]
]
]

(* {Hold[


we see that LoadJavaClass comes back unevaluated. You have to be careful here because when you don't Hold the return values, it can happen that things are evaluated at the main kernel and you think they work properly. So if you accidentally loaded JLink on the main kernel, you think the class loading works.

After some more debugging, it seems the following works

LaunchKernels[2]

ParallelEvaluate[
myMemoryConstraint = 9560566784;
JLinkInstallJava[];
With[{currentMemory =
javalangmanagementManagementFactorygetOperatingSystemMXBean[]@getFreePhysicalMemorySize[]},
If[currentMemory < myMemoryConstraint, Quit[]]
], 1]
]


The problem with using Quit[] is that you don't know when your kernels are closed. Mathematica will only complain when you try to give them something to do and it notices that the links are dead. Therefore, I get the following behavior

ParallelEvaluate[1]
(* {1, 1} *)

mem = RandomInteger[100, {500, 500, 500}];

ParallelEvaluate[1]


-
Thank you halirutan! I tried your code. But it seems that it doesn't work. Try ParallelTable[tt[i] = RandomInteger[100, {500, 500, 500}], {i, 1, 2}], you will see that the kernels are closed at first, then relaunched automatically and evaluate well give the result regardless of the memory –  matheorem Dec 14 '13 at 2:05
@matheorem Can you give details about the system you use? Operating system, Mathematica version? –  halirutan Dec 14 '13 at 2:19
@matheorem I guess at this point you have to think about how to cancel the calculation without the subkernel being restarted. Nevertheless the overall procedure, testing memory and doing something if it exceeds, works. You might try to use lowlevel link-communication to return from the subkernel. –  halirutan Dec 14 '13 at 3:10
I currently use Linux and Mathematica 8. I don't know how to use lowlevel link-communication to do this. But I found that even if you don't evaluate ParallelTable expression. Just a normal non-parallel expression like RandomInteger[100, {500, 500, 500}] for second time, the memory will still not get controlled. –  matheorem Dec 14 '13 at 10:12
@matheorem Have you seen Evaluation -> Parallel Kernel Configuration? There you can specify the behaviour when I subkernel crashes. It seems to be relaunch on default. Maybe this helps. –  halirutan Dec 16 '13 at 1:32