I have a set of data (about 20 million datapoints for my real data) of the form {x, y, z, m}
. There are 3 spatial coordinates x
, y
and z
and a fourth value, the mass (m
) of the element at that point.
I want to bin the data onto a integer spaced 3D grid. And then bin the masses into a series of specific ranges. I want the number of masses that fall into each mass range for each point on the grid. I guess, for n
ranges I will need n
3D arrays or alternatively a 3D array where each point has a list of n
values.
Let's start with some random data
coords = RandomReal[{-100, 100}, {10^5, 3}]; (*these can be negative*)
masses = RandomReal[400, 10^5]; (*these can't be negative*)
and some random ranges to bin the masses, we'll make 50 as this is about how many I have in my real data:
ranges = Partition[Union[Round[RandomReal[400, 100], 0.001]], 2];
Based on this related question: https://stackoverflow.com/questions/8178714/mathematica-fast-2d-binning-algorithm. I started by rounding the coordinates, I want to bin onto unit bins so I simply used Floor
on the coordinates.
indices = Floor[coords];
and in my case the masses have more precision than the mass range specifications so lets round those now too.
roundedmass = Round[masses, 0.001];
Now based on the related question I used GatherBy
to collect like coordinates into lists.
gathered = GatherBy[Transpose[{indices, roundedmass}], First];
However, I'm stuck as to how to quickly bin the masses in each of the gathered coordinate-bins into the various mass ranges. In the end I guess I will have 50 3D arrays where each coordinate-bin has a value corresponding to the number of masses in the specified mass-range. 50 mass ranges = 50 3D arrays (probably sparse as there will be lots of empty space), or alternatively a 3D array where each point has a list of 50 values.
I've tried to use BinCounts
but this also bins in between the mass ranges so I had to drop ever other bin that was returned, seems wasteful...:
fbinmasses = BinCounts[#, {Flatten@ranges}][[;; ;; 2]] &
binnedmass = ParallelMap[fbinmasses, gathered[[All, All, 2]]];
combinedoutput = Transpose[{gathered[[All, 1, 1]], binnedmass}];
sortedoutput = Sort[combinedoutput, OrderedQ[{#1[[1]], #2[[1]]}] &];
Now, I would need to make a sparse array, maybe like this (going to have issues with negative inidices...? And it seems like you can't store a list in a sparse array.
output = SparseArray[gathered[[All, 1, 1]] -> binnedmass]
binnedmasses
should bebinnedmass
in thecombinedoutput
calculation $\endgroup$