I have a matrix of data ~(70,000x70,000). I am searching connection between these data points. I use euclidean distance for that purpose. If it is less than my threshold (that means they have connection) I put 1 into adjacency matrix. If it is greater than my threshold I put 0. I need to change elements as following.
import data file data size n For[i = 1, i <= n, i++, For[j = 1, j < i, j++, eucdis[[i, j]] = N[EuclideanDistance[data[[i, All]], data[[j, All]]]]; If [eucdis[[i, j]] <= epsilon , rp[[i, j]] = 1; rp[[j, i]] = 1; ]; ]; rp[[i, i]] = 1; ];
Memory usage gets 100% and PC gets frozen.
Is it possible to limit memory usage, so it can take longer but could be done, or any other method to finish this calculation?