3
$\begingroup$

I am trying to run a code that gives out a matrix but, I seem to encounter some memory issues while trying to run it. The code is:

rm[{mx_, mz_}] := {0.048 mx (1 - 1/(2 Abs[mx])), 0.048 mz (1 - 1/(2 Abs[mz]))}; 

ListOfm = Outer[{#1, #2} &, DeleteCases[ Range[-128/2, 128/2, 1], 0, Infinity], DeleteCases[ Range[-128/2, 128/2, 1], 0, Infinity], 1];
ListOfrm = Map[rm, ListOfm, {2}];

rn[{nx_, nz_}] := {0.016 nx (1 - 1/(2 Abs[nx])), 0.016 nz (1 - 1/(2 Abs[nz]))} 

ListOfn = Outer[{#1, #2} &, DeleteCases[Range[-384/2, 384/2, 1], 0, Infinity], DeleteCases[Range[-384/2, 384/2, 1], 0, Infinity], 1];
ListOfrn = Map[rn, ListOfn, {2}];



Fun = Compile[{{rm, _Real, 1}, {rn, _Real, 1}},

      Exp[-((Compile`GetElement[rm, 1] - Compile`GetElement[rn, 1])^2/2 + 
            (Compile`GetElement[rm, 2] - Compile`GetElement[rn, 2])^2/2)], 

      CompilationTarget -> "C", Parallelization -> True, RuntimeAttributes -> {Listable},
      RuntimeOptions -> "Speed"];

 With[{Fun= Fun},
      Mat = Compile[{{rm, _Real, 2}, {rn, _Real, 2}},

                    Table[Table[Fun[Compile`GetElement[rm, i], Compile`GetElement[rn, j]],
                               {i, 1, Length[rm]}], {j, 1, Length[rn]}], 

            CompilationTarget -> "C", Parallelization -> True, 
            RuntimeAttributes -> {Listable}, RuntimeOptions -> "Speed"]];

When I input

Mat[Flatten[ListOfrm, 1], Flatten[ListOfrn, 1]]

I receive an error that

General::nomem: The current computation was aborted because there was insufficient memory available to complete the computation.

Throw::sysexc: Uncaught SystemException returned to top level. Can be caught with Catch[[Ellipsis], _SystemException].

Out[124]= SystemException["MemoryAllocationFailure"]

However if I use lower number of elements (64 and 192) like:

ListOfm = Outer[{#1, #2} &, DeleteCases[ Range[-64/2, 64/2, 1], 0, Infinity], DeleteCases[ Range[-64/2, 64/2, 1], 0, Infinity], 1];
ListOfrm = Map[rm, ListOfm, {2}];

 ListOfn = Outer[{#1, #2} &, DeleteCases[Range[-192/2, 192/2, 1], 0, Infinity], DeleteCases[Range[-192/2, 192/2, 1], 0, Infinity], 1];
ListOfrn = Map[rn, ListOfn, {2}];

the error does not show up but the computation takes a lot of time.

I want to use higher number of elements with a good time efficiency. How can I deal with this error and speed up my code?.

$\endgroup$
  • $\begingroup$ The $116384 \times 147456$ matrix that you try to assemble would require more than 147 GB of RAM. So either you have to buy a cosiderable amount of RAM or you have to think about another way to accomplish your actual goal. $\endgroup$ – Henrik Schumacher Sep 23 '18 at 22:21
  • $\begingroup$ If I give a minimum value below which the function evaluates to zero then can I make a sparse matrix out of it?. $\endgroup$ – jsid Sep 23 '18 at 22:44
3
$\begingroup$

Did you notice that your code doesn't run in parallel? Compiled parallelization works by using the Listability of Mathematica and then your code is run in parallel. To give a short pseudo-example: Instead of doing this

f[list_] := Table[DoSomething[element], {element, list}];
f[myList]

you will implement it like that

f[element_] := DoSomething[element];
f[myList]

So you define your compiled function for one element you want to process and you do not iterate over the list of elements yourself. To make it perfectly clear, look at this minimal example

fc = Compile[{{elm, _Real, 0}}, elm + 1, 
  RuntimeAttributes -> {Listable}]
fc[{1, 2, 3, 4}]
(* {2., 3., 4., 5.} *)

If you understand this then you will see why your code does not run in parallel. Let us fix this by using a definition of fun that works and replaces your nested Table in Mat

funParallel = Compile[{{rm, _Real, 2}, {rn, _Real, 1}},
   Table[
    Exp[-((Compile`GetElement[rm, i, 1] - 
            Compile`GetElement[rn, 1])^2/
         2 + (Compile`GetElement[rm, i, 2] - 
            Compile`GetElement[rn, 2])^2/2)],
    {i, Length[rm]}
    ],
   CompilationTarget -> "C", Parallelization -> True, 
   RuntimeAttributes -> {Listable}];

Look carefully at the type-specs in my Compile call. We will take the whole matrix for rm, but only a single point for rn. That means, inside the body, we actively iterate over the rm, but when we provide a matrix of rn, then this will happen in parallel.

I have made some tests with smaller values and the result is the same as for your Mat call. However, it runs faster and doesn't use so much memory on my machine

r2 = funParallel[Flatten[ListOfm, 1], Flatten[ListOfn, 1]]; // AbsoluteTiming

takes about 20 seconds here and the result is 18GB large. I'm not sure I missed something, because my result has dimensions {16384, 147456} which is substantially smaller than what Henrik said, but maybe he only included an additional 1 accidentally.

$\endgroup$
  • $\begingroup$ The $1$ was only a typo. My real fault was to forget that to divide by $8$, so the estimated size is 147 GigaBit which is 18 GigaByte. Silly me. Good job btw (+1). $\endgroup$ – Henrik Schumacher Sep 24 '18 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.