Selecting certain rows of two huge matrices

I have two huge matrices m1 and m2 (length 25000), and for simplicity let us assume they are:

m1 = {{1, 1, -56}, {1.3, 2, 3.06}, {2, 0, -30.02}, {3,
1, -7.291}, {3.5, 2, 1.93}, {4, 0, 0}, {5, 0, 0}, {5.5,
1, -356.4}, {6, 1, 9.945}, {7, 0, -7.512}};
m2 = {{1, 1, -56}, {1, 2, 3.06} + .2, {2,
0, -30.02}, {3, 1, -7.291} + .3, {3, 2, 1.93}, {4, 0, 0}, {5, 0,
0}, {5, 1, -356.4} + .4, {6, 1, 9.945}, {7, 0, -7.512}};


I wrote the following code with the help of this forum to pick those rows of both matrices that have the same first value. However, the code is very slow

picker1 = m1[[All, 1]];
picker2 = m2[[All, 1]];
newM1 = Cases[m1, x_ /; MemberQ[picker2, x[[1]]]]
newM2 = Cases[m2, x_ /; MemberQ[picker1, x[[1]]]]


I know the code is not an efficient and I appreciate if you can help me with more efficient and fast code.

• Pick[m1, MemberQ[picker2, #] & /@ picker1] and Pick[m2, MemberQ[picker1, #] & /@ picker2]?
– kglr
Apr 26, 2018 at 22:17
• qahtah, your application could be boosted immensely if you can achieve to have matrices with either all integers or all real numbers, so that they can be packed. However, MemberQ has certain problems with floating point numbers. Do the first entries in each row in your huge matrices happen to be integers? Apr 26, 2018 at 22:25
• @HenrikSchumacher no they are real. Apr 26, 2018 at 23:44
• @kglr too slow. Apr 26, 2018 at 23:46

Using MemberQ is not a good idea, performancewise.

Let's create some example data

n = 25000;
m = 1000;
p1 = RandomInteger[{1, m}, n];
p2 = RandomInteger[{1, m}, n];
a1 = RandomReal[{-1, 1}, {n, 2}];
a2 = RandomReal[{-1, 1}, {n, 2}];


Note that we separated the "pickers" p1 and p2 from the data a1, a2. So m1[[i]] would correspond to Join[{p1[[i]]},a1[[i]]]. This way, p1, p2, a1, and a2 are all packed arrays.

For each list of pickers p1, p2, I create an Association with values all 1.

dict1 = AssociationThread[p1, 1];


This way, dict1[x] will return 1 (read as True) if x is a member of p1 and Missing["KeyAbsent", x] otherwise. Lookup can convert Missing[...] to a default value; we use 0 (read as False) in the following. We create new lists of pickers newp1 and newp2 along with the new data matrices newa1 and newa2:

With[{b1 = DeveloperToPackedArray[Lookup[dict2, p1, 0]]},
newp1 = Pick[p1, b1, 1];
newa1 = Pick[a1, b1, 1];
]; // AbsoluteTiming
With[{b2 = DeveloperToPackedArray[Lookup[dict1, p2, 0]]},
newp2 = Pick[p2, b2, 1];
newa2 = Pick[a2, b2, 1];
]; // AbsoluteTiming


0.002823

0.002927

Here, we use Pick with 1 as third argument to pick all rows from a1 for which the corresponding entry in b1 is 1. Doing the anologous by constructions involving MemberQ takes about 5 seconds on my machine.

One has to be careful if the pickers are floating point numbers: This construction with Association might break down due to rounding errors since Associations are very literal about the keys. But that's the same with MemberQ.

• Thanks. The first values of rows are distinct. It works very well. Apr 27, 2018 at 0:34