I have a large set of points in 3D and I'm trying to identify all the points that lie within a certain distance of each point. Then using this data store the vector between the pairs of points.
I have about 1M points and this results in millions of vectors.
It seems that when I Map
Nearest
over my set of points it is unpacking the array (because it becomes ragged?) and this appears to take up all my memory. At the end I Flatten
the array to a specific depth and can manually pack it again and I estimate it should fit in memory.
closepointsvectors
finds all the points within 1 unit of point x and then returns the vector between the point x and the "close" points. I then Map
this over all the points in the dataset. This generates many many vectors.
Here is a minimal (non)working example.
data = RandomReal[10, {10^6, 3}];
nearestfunction = Nearest[data];
closepointsvectors[x_] := (x - #) & /@ nearestfunction[x, {Infinity, 1}]
(*lets try on a small subset of the data*)
smalltest = Flatten[Map[closepointsvectors, data[[1 ;; 10000]]],2]; // AbsoluteTiming
(*{1.5870908, Null}*)
Developer`PackedArrayQ@data
(*True*)
Developer`PackedArrayQ@smalltest
(*False*)
ByteCount[smalltest]
(*40818664*)
packedsmalltest = Developer`ToPackedArray@smalltest;
Developer`PackedArrayQ@packedsmalltest
(*True*)
ByteCount[packedsmalltest]
(*10204824*)
Is there a way to reduce memory usage, possibly keeping the array packed the whole time. Can I pad the array with zeroes as I go to prevent it from becoming ragged? I tried to take advantage of compile but it doesn't appear to work with Nearest
.
nearestfunction
? $\endgroup$