I have a Mathematica notebook that I have been running on my desktop and am working on moving it over to a high-performance cluster.

One part of the code is a random walk algorithm that uses a matrix "landscape" to determine probabilities at each step. The random walk algorithm will be run millions of times.

I'd like to take full advantage of parallelization by running the random walk separately from the part of the notebook that creates the matrix. The random walks all need to be using the same landscape matrix.

My question is, is it possible to put the matrix in an environment variable that is then shared by the algorithm instances that I launch?

The supercluster is using slurm to manage jobs, so the idea would be to run the script that makes the matrix, then run the random walk algorithm over and over. This would allow me to make full use of the cluster since each instance could access its own cores and ram.

  • $\begingroup$ Is the matrix updated or just they all run on the same matrix independently? If the latter, I'd think the simplest approach would be to write it to a file. $\endgroup$ – Pillsy May 1 '20 at 16:27
  • $\begingroup$ Why don't you just use the parallel tools (ParallelTable, ParallelMap, etc.) which take care of sharing the data (DistributeDefinitions)? $\endgroup$ – Szabolcs May 1 '20 at 16:32
  • $\begingroup$ Pillsy, they all run on the same matrix independently, but if I put it in a file, then won't it have to be loaded into memory every time the algorithm runs on it? It's 10^4 cells square. $\endgroup$ – Joseph Thiebes May 2 '20 at 9:58
  • $\begingroup$ Szabolcs, I didn't know about DistributeDefinitions before. Will that be the best tool to share the matrix with each of the kernels? $\endgroup$ – Joseph Thiebes May 2 '20 at 9:59