I found the performance of built-in function
MemoryConstrained is very poor. It stops the evaluation after the memory limit has already been exceeded greatly than the values I set.
There are several other questions on how to constrain memory usage during an evaluation. For example, this one. But the method seems not working for parallel situation(for example, if you perform
MemoryInUse is specific to each kernel. No bother, there is another way, that is use Jlink to get the system's free physical memory information( see here ). So I combined these two method. Reach the following code:
Needs["JLink`"]; InstallJava; LoadJavaClass["java.lang.management.ManagementFactory"]; intervalBetweenTests = 1; RunScheduledTask[If[( JavaBlock[ java`lang`management`ManagementFactory`getOperatingSystemMXBean\ @# &@getFreePhysicalMemorySize]) < 1*1024^3, Quit], intervalBetweenTests]
After running this code, mathematica launch a watchdog, and any evaluation (no matter parallel or not) will be aborted if the free physical memory in current machine is below 1GB.
The code just mentioned works on a single machine. But I have to work on HPC, every parallel evaluation is performed on several different machine simultaneously, and I have to make sure that free physical memory won't below a threshold on every machine I use during a parallel evaluation, otherwise the machine will be dead, the admin will be very unhappy and mad about me:)
ParallelEvaluate the above code, intended to launch watchdogs on all available kernels. That is
ParallelEvaluate[Needs["JLink`"]; InstallJava; LoadJavaClass["java.lang.management.ManagementFactory"]; intervalBetweenTests = 1; RunScheduledTask[If[( JavaBlock[ java`lang`management`ManagementFactory`getOperatingSystemMXBean\ @# &@getFreePhysicalMemorySize]) < 1*1024^3, Quit], intervalBetweenTests]]
but didn't work. The above code output a list of same ScheduledTask object, and evaluate
ScheduledTasks shows no running scheduledtask at all. And of course the The free physical memory is not constrained as expected when doing calculation.
- it seems that
RunScheduledTaskcan't been parallelized, is that true?
- why built-in function
MemoryConstrainedis so poor?
- finally, how to achieve memory control across machines in mathemaica?
as OleksandrR said, an HPC usually equipped with a job management system, it provide resource control of the job. for example in PBS job system,
pmem can constrain the memory usage per process of a job. But to constrain memory usage inside mathematica itself is much more adaptive. There are cases
- some small cluster doesn't equipped with a job management system
- the way to use the HPC is variable. Very often we can use mathemaica interactively on HPC by submitting an interactive PBS job, and if the pmem is set at the beginning, then all the calculations doing in this interactive job is restricted per process, while we should have been able to use more memory if we didn't set the pmem when submit the job.
and other cases you can think of, but I think these two is enough.