# Memory use for large computation

I am running a "big" program on my 32G RAM laptop. When MMA is running, roughly there is about 28G RAM "available" to use.

So I did a bit of a test.

mmu = MaxMemoryUsed[];
n = 5000;
(ans = ParallelTable[ myfunc[], n];) // AbsoluteTiming
MaxMemoryUsed[] - mmu


This returns

16362968

Assuming that I need to run 10 000 000 times, does that mean that I need at least 200 X 16362968 of memory?

Unfortunately on my hard-disk, my C drive is running low, but I am happy to use D drive for paging. If I set the page file size to 50GB, and enable in MMA,

Needs["JLink"]
ReinstallJava[JVMArguments -> "-Xmx51200m"]


Should this be enough to run it?

• I'm not sure you'll need that much memory, probably slightly less (you can try to run the benchmark for different n to see how it scales). Regarding the memory cap: I don't think you need to change anything to allow MMA to use more RAM. The code you've posted only increases the cap of the JLink process, but that's not the kernel (JLink is mainly used for the documentation center and frontend, unless of course you're calling Java functions). The kernel, as a normal executable (i.e. not jar) has no memory limit as far as I know – Lukas Lang Jun 10 '18 at 14:17
• @Mathe172 oh, I did try to just run it, but then in the end I got a warning saying not enough memory!!! Does that mean no way I can do it ???? I can run 1 million without any difficulty, but I now want to try 10 million for example. – Chen Stats Yu Jun 10 '18 at 14:30
• This is not enough info. But apparently, you don't need the whole list ans in memory. Just divide the ParallelTable into chunks and safe the results to file. Moreover, myfunc needs a fixed amount of memory per kernel as "scratch pad". This memory can be recycled in each call (best way would be to program it so that the memory gets overwritten and not freed and reallocated). This memory is also included in the 16362968 Byte that you measured with MaxMemoryUsed`. So my prediction is that you will need much less than the 200-fold amount of memory. – Henrik Schumacher Jun 11 '18 at 6:49
• At least macOS handles the swap file automatically and dynamically (so there is no need of assigning an upper bound. There seem to be a tools availabe for other unixoid OSes that accomplish that, too. But don't ask me about Windows. – Henrik Schumacher Jun 11 '18 at 6:54
• @ChenStatsYu How did the error look like? I've told MMA to use up >64GB of ram on my 16GB PC - the only thing that happened was that it died due to everything being pushed to the HDD. So there seems to be nothing inherently limiting the memory that MMA can use – Lukas Lang Jun 13 '18 at 9:40