Making faster extraction of table elements in Mathematica or beyond

Question

Consider the following code:

indextype = 1;
indexE = 2;
testcode = 
  Hold@Compile[{{data, _Real, 2}, {Nsampled, _Integer}}, 
       Module[{n1 = 0, n2 = 0, len = Length[data], datanew1, 
         type1 = 1., type2 = 0., e1 = 0., e2 = 0.},
        datanew1 = data;
        Do[
         (*Indices of particles forming a random pair*)
         n1 = RandomInteger[{1, len}];
         n2 = n1;
         While[n2 == n1, n2 = RandomInteger[{1, len}]];
         (*Extracting types of particles*)
         
         type1 = (2 RandomInteger[{0, 1}] - 1)*
           If[n1 <= len/2,(*datanew1[[n1,indextype]]*)
            Compile`GetElement[datanew1, n1, indextype], 4.];
         
         type2 = (2 RandomInteger[{0, 1}] - 1)*
           If[n2 <= len/2,(*datanew1[[n2,indextype]]*)
            Compile`GetElement[datanew1, n2, indextype], 4.];
         e1 = Compile`GetElement[datanew1, n1, indexE];
         e2 = Compile`GetElement[datanew1, n2, indexE];
         (*If e1, e2, type1, type2 satisfy some rare condition, 
         proceed to the rest of the code, modify datanew1, etc.*)
         , {i, 1, Nsampled, 1}];
        datanew1
        ], CompilationTarget -> "C", RuntimeOptions -> "Speed"] /. 
     OwnValues@indextype /. OwnValues@indexE // ReleaseHold;

It iteratively extracts the components of the two rows of data, and then is supposed to perform some operation in a very rare case when the components satisfy some condition. The operation will then modify data (this is why I have made its copy called datanew1), including changing its size.

It turns out that for my study, the main timing of the routine is due to this do-nothing extraction - it is 95% or so. I provide the relevant toy dataset example below:

npts = 3000;
ncells = 10^3;
data = Table[RandomReal[{0, 1}, {npts, 2}], ncells];
ParallelMap[testcode[#, 0.1*npts^2/2] &, data]; // RepeatedTiming

I cannot change the amount of the do-nothing extraction. Because of this, I am interested in increasing the speed of this extraction.

Is it possible to do it somehow in Mathematica? Or (this is to C++ experts), maybe it may make sense to do it in C++ and then use LibraryLink?

Edit

I am simulating the dynamics of some physical system made of particles. I represent each particle by a row of a table. I split the system into sub-systems (this is what each element of data means, with each row characterizing a particle). The dynamics are via binary interactions between particles. First, I randomly select a pair of particles and extract their properties (this is what the code above does). Second, I compute some conditions, and if they are satisfied, I simulate the interactions and modify the initial table by modifying, deleting, or adding the two rows of data (in practice, I pre-allocate the large storage of data and do not delete/add rows but rather move them from active to inactive rows or vice versa).

Answer 1

It's hard to give a precise answer, because the OP hasn't asked a precise question.

Also, what the testcode returns equals to data.

res1=ParallelMap[testcode[#,0.1*npts^2/2]&,data];//AbsoluteTiming

{70.0578, Null}

res2=Map[testcode[#,0.1*npts^2/2]&,data];//AbsoluteTiming

{56.1665, Null}

res1 == res2 == data

True

You could trim a couple more seconds compiling RuntimeAttributes -> {Listable} instead of using Map.

But, those are minor improvements.

From what I see inside of testcode, you employ Do[] to generate 450,000 random pairs {n1,n2}, so that n1 != n2, and then generate 450,000 random type1 and type2, and then extract e1 and e2 from data using n1 and n2 as position indexes.

You could turn this whole function in vectorized fashion.

Considers this:

randomPairs=Take[DeleteCases[RandomInteger[Length[data]-1,{Ceiling[0.1*npts^2/2*1.1],2}]+1,Transpose[{r=Range@Length@data,r}]],0.1*npts^2/2];//RepeatedTiming

Gives 450,000 random pairs of {n1,n2}, such that n1!=n2, in just 0.2 seconds.

{0.207896, Null}

If you proceed in this manner to generating type1 and type2 lists, you could end up extracting all necessary e1s and e2s at once, in the data[[{e11,e12,e13..e1450000}]] manner, in just seconds.

Hope this helps.