Using Arnoldi method to find some eigenvalues (say 20), with shift value taken in the middle of the spectrum, of a large sparse matrix the Kernel crashes with no error returned. The crash occurs if and only if the size of the matrix exceeds approximately 100,000 by 100,000 with about 1,500,000 specified elements. The machine is an Intel Xeon with 64GB of RAM running CentOS, the whole matrix occupies only 35MB as given by ByteCount[]. Could it be an issue of memory mismanagement by the linear solver invoked by Arnoldi? A real-time monitoring of the memory used by the machine does not show that it runs out of memory. Code follows.

sx = SparseArray[{{0, 1/2}, {1/2, 0}}];
sy = SparseArray[{{0, I/2}, {-I/2, 0}}];
sz = SparseArray[{{1/2, 0}, {0, -1/2}}];
id = SparseArray[{{1, 0}, {0, 1}}];
L = 20;
U = 1.1;
htb = RandomReal[{-1/U, 1/U}, L];
Jtb = 1/(2 (1 - htb^2));
(*xx part*)
Hxx = Sum[
 Jtb[[i]] KroneckerProduct @@ 
   Table[If[Or[n == i, n == i + 1], sx, id], {n, 1, L}], {i, 1, 
  L - 1}] + 
Jtb[[L]] KroneckerProduct @@ 
  Table[If[Or[n == 1, n == L], sx, id], {n, 1, L}] + 
Sum[Jtb[[i]] KroneckerProduct @@ 
   Table[If[Or[n == i, n == i + 1], sy, id], {n, 1, L}], {i, 1, 
  L - 1}] + 
Jtb[[L]] KroneckerProduct @@ 
  Table[If[Or[n == 1, n == L], sy, id], {n, 1, L}]; // Timing
(*zz part*)
Hzz = Sum[
 Jtb[[i]] KroneckerProduct @@ 
   Table[If[Or[n == i, n == i + 1], sz, id], {n, 1, L}], {i, 1, 
  L - 1}] + 
Jtb[[L]] KroneckerProduct @@ 
  Table[If[Or[n == 1, n == L], sz, id], {n, 1, L}]; // Timing
SZt = Sum[KroneckerProduct @@ Table[If[n == i, sz, id], {n, 1, L}], {i, 1, 
halfs = Flatten[Position[Normal[Diagonal[SZt]], 0]];
Htot = Hxx + Hzz;
Hhs = Htot[[halfs, halfs]];
Hxx =.;
Hzz =.;
Htot =.;
\[Mu] = Tr[N[Hhs]]/Length[halfs];
{evl, eve} = Eigensystem[N[Hhs], -20, Method -> {"Arnoldi", "Shift" -> \[Mu]}];
  • 3
    $\begingroup$ This should probably be reported to support. Here, we can't do much unless you can upload your sparse matrix somewhere. $\endgroup$ – J. M.'s technical difficulties Nov 15 '17 at 14:11
  • $\begingroup$ It would be nice to see the code you used for invoking the Arnoldi method and either the sparse matrix or (even better) the code for generating it, if possible. $\endgroup$ – Henrik Schumacher Nov 15 '17 at 14:57
  • $\begingroup$ I've added the code in the edit. I am sure there's a simpler version of this which makes it crash, as it appears to be dictated by the size of the matrix. Running the same code for L=18 does NOT crash the kernel. But maybe J.M. is right, this is a subject to be reported to support. $\endgroup$ – Antonello Scardicchio Nov 16 '17 at 14:55
  • 1
    $\begingroup$ Running this in a debugging kernel indicates it really does just run out of memory. I do not know where the memory grab happens though. Prior to invoking Eigensystem I see a MaxMemoryUsed[] result of around 3.4 GB and my machine has around 8 GB of RAM>. I will surmise that the array gets copied once and that some large amount of space is allocated for intermediate computations and for the result. But that's just a wild guess. $\endgroup$ – Daniel Lichtblau Nov 16 '17 at 17:28
  • $\begingroup$ Thank you Daniel. Eigensystem with shift-invert Arnoldi method calls a linear solver which, if written inefficiently, might be the culprit. However 64GB is a lot to be filled just for this task... $\endgroup$ – Antonello Scardicchio Nov 18 '17 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.