I have a matrix of data ~(70,000x70,000). I am searching connection between these data points. I use euclidean distance for that purpose. If it is less than my threshold (that means they have connection) I put 1 into adjacency matrix. If it is greater than my threshold I put 0. I need to change elements as following.
import data file
data size n
For[i = 1, i <= n, i++,
For[j = 1, j < i, j++,
eucdis[[i, j]] =
N[EuclideanDistance[data[[i, All]], data[[j, All]]]];
If [eucdis[[i, j]] <= epsilon ,
rp[[i, j]] = 1; rp[[j, i]] = 1;
];
]; rp[[i, i]] = 1;
];
Memory usage gets 100% and PC gets frozen.
Is it possible to limit memory usage, so it can take longer but could be done, or any other method to finish this calculation?
Sow
andReap
to keep track of all "close" connections, assuming each entry on averate has but few. Then create aSparseArray
from the set of keepers. $\endgroup$