2
$\begingroup$

I am working on becoming more and more familiar with MMA and on a scale from one to ten, I would rate myself as a four or a five. I understand vectorization and that loops-are-very-bad. I have recently learned anonymous functions and mapping in MMA. I have yet to understand patterns and associations so I am avoiding them for right now.

This is my first non-trivial program in MMA. I have optimized it as much as I can. But I would like to see if it can be made even faster.

The problem description is that I have some particles and some measurements associated with them. The particles occupy integer coordinates. The occupied coordinates can have one or more particles. I want to specify a threshold and then pick coordinates which have more than the threshold number of particles and then export them to a text file.

  • The main goal here is time efficiency. Can this code be made faster? I don't care about memory efficiency.
  • I am using as many existing functions as I could find. Are there any others that I missed?
  • The actual problem is much larger and is repeated many times. data contains about 50,000 particles. threshold is at ten. Out of those 50,000 xyzCoordinates, only 1500 coordinates are exported, which corresponds to about half (25,000) of the particles.
  • I am trying to learn good MMA coding habits. Any general comments on my style/formatting are also welcome.
  • The exporting to a text file is being done simply because sometimes I just want to open up a file and see the particle data. It is not super important so I am willing to be convinced if someone makes a good argument against it. Maybe there is another exporting command/formatting which is much faster? Like the .mat files in MATLAB?
  • No matter what the format, the exporting has to be done though, because I have to post-process the separated particles. Profiling tells me that exporting to CSV accounts for about 25% of the time. The rest of it is taken up by the Take command inside the Table. The rest of the code is negligible but if it can be improved in anyway, I will love to learn.

Here is the explanation of the code with a complete minimum working example underneath it.

In[1]:= (*Seeding the RNG for consistency.*)
SeedRandom[1234];

In[2]:= (*Creating fake data with a hundred rows and seven columns. \
Each row represents measurements for a particle so there a hundred \
particles here.*)
data = RandomReal[1, {100, 7}];

In[3]:= (*These are the (x,y,z) coordinates for each of the hundred \
particles. The coordinates are always positive integers, on a bounded \
integer lattice. In this case, x, y, and z each take values from \
{1,2,3}. Coordinates can have any number of particles, one or more.*)


xyzCoordinates = RandomInteger[3, {100, 3}];

In[4]:= (*Now we tally the xyzCoordinates to count how many particles \
are there at each coordinate.*)
Take[Tally@xyzCoordinates, 10]

Out[4]= {{{0, 1, 1}, 2}, {{2, 0, 1}, 2}, {{0, 1, 2}, 1}, {{2, 0, 2}, 
  4}, {{0, 0, 2}, 3}, {{2, 2, 3}, 6}, {{2, 3, 2}, 4}, {{1, 2, 0}, 
  1}, {{3, 1, 2}, 3}, {{0, 3, 1}, 5}}

In[5]:= (*We define a threshold here. This will be a small integer, \
let's say between one and ten.*)
threshold = 2;

In[6]:= (*Now we select the xyzCoodinates which have more than \
threshold=2 particles.*)
coordinatesToExport = 
 Select[Tally@xyzCoordinates, #[[2]] > threshold &][[All, 1]]

Out[6]= {{2, 0, 2}, {0, 0, 2}, {2, 2, 3}, {2, 3, 2}, {3, 1, 2}, {0, 3,
   1}, {1, 3, 2}, {0, 0, 3}, {0, 1, 3}, {3, 3, 3}, {2, 0, 3}, {1, 3, 
  0}, {2, 3, 0}, {1, 0, 0}}

In[7]:= (*Now we know how which xyzCoordinates have more than \
threshold=2 particles. For each of those selected coordinates, we \
select the corresponding rows from data and export to a text file. I \
have commented out the Export command but if you uncomment it, you \
will end up with fourteen small textfiles in the same directory as \
this notebook, each containing some number of particles' data.*)
Table[
  dataToWrite = 
    Pick[data, (# == coordinatesToExport[[fileNameIndex]]) & /@ 
      xyzCoordinates];
  (*outputFile=NotebookDirectory[]<>ToString[fileNameIndex]<>".txt";
  Export[outputFile,dataToWrite,"CSV"];*),
  {fileNameIndex, 1, Length[coordinatesToExport], 1}];

In[8]:= (*Here is the complete MINIMUM WORKING EXAMPLE*)
SeedRandom[1234];
data = RandomReal[1, {100, 7}];
xyzCoordinates = RandomInteger[3, {100, 3}];
threshold = 2;
coordinatesToExport = 
  Select[Tally@xyzCoordinates, #[[2]] > threshold &][[All, 1]];
Table[
  dataToWrite = 
    Pick[data, (# == coordinatesToExport[[fileNameIndex]]) & /@ 
      xyzCoordinates];
  (*outputFile=NotebookDirectory[]<>ToString[fileNameIndex]<>".txt";
  Export[outputFile,dataToWrite,"CSV"];*),
  {fileNameIndex, 1, Length[coordinatesToExport], 1}];
$\endgroup$
  • 1
    $\begingroup$ Pick[data, xyzCoordinates, Alternatives @@ coordinatesToExport] will generate the data to write in one go, s/b order or two magnitude faster. In any case, the writing of data will likely swamp any internal speedup. $\endgroup$ – ciao Sep 4 '19 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.