# Extracting matrix rows when one of its elements is part of a list (without a loop)

I have a matrix whose rows I want to extract based on whether the elements of one of its columns is a member of another vector (calling it the "comparison vector"). I would like to get better at using functional programming, and so I want to avoid using a loop, if it´s possible. I believe I have the right functions: Selectand MemberQ. But I can´t coerce MemberQ to compare the element of the matrix to each element of the comparison vector.

If there´s a duplicate answer, I´m willing to be lead to it (I´ve searched though)...

 mymatrix={{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {3, 1, -7.291}, {3, 2,1.93}, {4, 0, 0}, {5, 0, 0}, {5, 1, -356.4}, {6, 1, 9.945}, {7,0, -7.512}};

compvector={1,2,6,7,11,12,16,17};


I would like to extract the rows of ´mymatrix´ based on whether the values of the first column of ´mymatrix´ are in ´compvector´.

• I suppose you want something like this: mat = {{1, 2, 3}, {5, 4, 6}, {6, 20, 13}}; cmpvec = {10, 13, 33, 44}; Select[mat, Intersection[#, cmpvec] != {} &]
– Aky
Aug 9, 2013 at 13:57
• Hi @Aky. As I said to Anon, thanks for your response, but it´s not quite right, and that´s due to my lack of information since the beginning... I´m not looking for whether a value is found in any part of the row - I´m trying to test whether a specific part of the row belongs to the list. Aug 9, 2013 at 14:16
• @Aky could you briefly explain what ´!= {} &´ does in your code? Aug 9, 2013 at 14:26
• {} is just an empty list. So the expression checks if the result of the Intersection is a non-empty list (meaning the two argument lists have at least one common element, which is how I had initially understood your question).
– Aky
Aug 9, 2013 at 16:19

New solution

Cases[mymatrix, {x_, _, _} /; MemberQ[compvector, x]]


{{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}}

I do not take credit for this solution, someone posted this here before me but in the confusion over what the problem was that person (whose name I do not remember) deleted his answer :(

Here's another answer that does not use Cases.

First let's define what columns are required to be in compvector, first and second for example:

required = {True, True, False}


Then

Select[mymatrix, And @@ (MemberQ[compvector, #] & /@ Pick[#, required]) &]


{{1, 1, -56}, {1, 2, 3.06}, {6, 1, 9.945}}

If the requirement is just that a specific column should exist, this can obviously be made a lot simpler. See aky's answer.

The complaint for the first version was that if there are many columns the pattern would also be very long, {x_,_,_,_ ...: this is not really true, one could write a short pattern to match such a list.

Old solution, not what the OP wants

First define some test data:

comparison = Range


{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

matrix = RandomInteger[100, {10, 5}]


{{46, 51, 84, 49, 52}, {12, 22, 7, 51, 56}, {74, 61, 9, 23, 93}, {97, 0, 23, 87, 78}, {23, 29, 83, 68, 21}, {79, 1, 25, 13, 84}, {23, 85,
35, 83, 83}, {2, 29, 50, 22, 88}, {34, 61, 91, 84, 29}, {60, 51, 96,
48, 68}}

The test:

Select[matrix, Length[Intersection[comparison, #]] > 0 &]


{{12, 22, 7, 51, 56}, {74, 61, 9, 23, 93}, {79, 1, 25, 13, 84}, {2,
29, 50, 22, 88}}

• Hi @Anon. Thank you for your response, but it´s not quite right (my fault). I´m not looking for whether a value is found in any part of the row - I´m trying to test whether a specific part of the row belongs to the list. But I´m going to try to use this ´Intersection´ function... maybe it´s useful. Aug 9, 2013 at 14:14
• @Verde See update. Aug 9, 2013 at 14:28
• Yes, this did it... thank you. I wanted to ask something about your answer, though. If my matrix had many more columns, using the syntax ´{x_,,,_}´ inside ´Cases´ wouldn´t be efficient. Do you know of another way of solving my question without using ´Cases´ in this way? I was working with ´Cases´function as well, but never got it to work properly... Aug 9, 2013 at 14:38
• @Verde I added a different solution which hopefully works better for you. Aug 9, 2013 at 14:53
• What I meant by not being efficient is that, if I had 100 columns in my matrix for instance, I would have to write ´_´ 99 times in the syntax ´{x_,,,_}´. Maybe efficient isn´t the right word... Aug 9, 2013 at 14:56

I recommend using Pick for these things. It has already been used by others but the simplest form using Alternatives hasn't been shown:

Pick[#, #[[All, 1]], Alternatives @@ #2] &[mymatrix, compvector]


{{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}}

For greater speed especially with longer comvectors we can use a Dispatch table:

fast[m_, c_] := Pick[m, m[[All, 1]] /. Dispatch @ Thread[c -> True]]

fast[mymatrix, compvector]


{{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}}

Timings with some larger data:

mymatrix = RandomInteger[9999, {50000, 3}];
compvector = RandomChoice[Range@9999, 150];

(* the faster of Michael's functions *)
michael[m_, c_] := With[{nf = Nearest[c]},
Pick[m, # - First /@ nf /@ # &@m[[All, 1]], 0]
]

Cases[mymatrix, {x_, _, _} /; MemberQ[compvector, x]] // Timing // First
Pick[#, #[[All, 1]], Alternatives @@ #2] &[mymatrix, compvector] // Timing // First
michael[mymatrix, compvector]                         // Timing // First
fast[mymatrix, compvector]                            // Timing // First


3.447

0.905

0.265

0.047

A run-off with Michael's method on even larger data:

mymatrix = RandomInteger[99999, {500000, 3}];
compvector = RandomChoice[Range@99999, 15000];

michael[mymatrix, compvector] // Timing // First
fast[mymatrix, compvector]    // Timing // First


15.943

0.327

• The timings for V9: In the first "race" (the 50000-row mymatrix), the timings of the four were respectively 0.594460, 0.158027, 0.140691, 0.059914. In the run-off, they were 2.479786, 0.725754. The relative improvement of michael from V7 to V9 is almost 15X - wow. Bully for NearestFunction, I guess. Aug 10, 2013 at 0:05

Assuming I've (now) understood your question properly, it's a very simple problem that deserves a very simple answer:

Select[mymatrix, MemberQ[compvector, First@#] &]


Here are a couple of ways:

nf = Nearest[compvector];
Pick[mymatrix, # - First /@ nf /@ # &@ mymatrix[[All, 1]], 0]
(* {{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}} *)

Pick[mymatrix, Times @@@ Outer[Plus, mymatrix[[All, 1]], -compvector], 0]
(* {{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}} *)


This one below is pretty fast for small compvector, but the complexity is not very good. It starts to lose out to the first one (the faster one) with the length of compvector exceeds about 170, and to Mr.Wizard's fast when the length is above about 70.

Extract[mymatrix, Position[mymatrix[[All, 1]], Alternatives @@ compvector]]

• +1 for being primed by using Nearest recently. :-) Aug 9, 2013 at 23:24
• @Mr.Wizard I considered the binary search, but I left it for you, if you wish. For the sake of variety, I'm looking for a less Pick-y solution that's as fast or faster. Aug 9, 2013 at 23:29
• Michael, I added an answer with my own preference (Alternatives). I believe that this function was improved some time after v7; could you please run the timings yourself, and also with compvector = RandomChoice[Range@9999, 2500]; and tell me what you get? Aug 9, 2013 at 23:34
• Michael, I added a much faster method using Dispatch. I'm still curious to know how Alternatives compares in later versions. Aug 9, 2013 at 23:52

Another possibility:

Pick [#, MemberQ[compvector, #] & @@@ #] &@mymatrix


=> {{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}}

Modifying Anon's first solution gives:

Cases[mymatrix, x_List /; MemberQ[compvector, x[]]]


As we can see there is no need to type {x_,_,_,_ ...} which is inconvenient for a matrix with a large number of columns.

{{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}}

• x_List in this code can be replaced with x_. There is no need to specify the List head. Aug 9, 2013 at 19:35
• In fact this is the modification of Hector's deleted post. Thanks to Hector. Aug 10, 2013 at 5:46
• @Mr.Wizard How many posts am I allowed to edit per day? Aug 13, 2013 at 11:36
• I'm going to say six (6). This is based on Area 51 saying we get 12 questions a day; I figure one person's edits should not be more than half that. We did just approve a lot more than that but it was a one-time thing because of the effort I know you put out. Aug 13, 2013 at 11:51
• @Mr.Wizard Ok. I am sorry for any inconvenience I may have caused. Aug 13, 2013 at 12:00

Modified post

I erased my post because I thought that the other solutions were better for the wording of your question. Your clarifications, especially the 100 column one, suggest that you need:

matrix = {{a, b, c}, {d, e, f}, {g, h, i}};
picker = RandomChoice[{b, h, u}, 30000000];
Timing[Cases[matrix, x_ /; MemberQ[picker, x[]]];]


Interestingly, this calculation is fast even when picker is a long vector.

Deleted post

It seems that you want something like

matrix = {{a, b, c}, {d, e, f}, {g, h, i}}; picker = {b, h, u};
Cases[matrix, {_, x_, _} /; MemberQ[picker, x]]


which returns {{a, b, c}, {g, h, i}}. Those are the rows whose second element is a member of that other vector picker.

• The first solution (the deleted post) is the same as Anon's first solution and the second solution (the modified post) is the same as mine. The only difference is that you have compared the second element of the lists. Aug 10, 2013 at 3:50
• @M6299: Anon refers to "someone posted this … that person … deleted his answer". Anon read my post in the 10 minutes it was alive. My modified post addresses Verde's the concern about not writing 99 Blank[]s. I went further and showed with Timing that such code is efficient. Aug 10, 2013 at 5:17
• I added that my post is the modification of your deleted post. Aug 10, 2013 at 5:48
list =
{{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {3, 1, -7.291}, {3, 2, 1.93}, {4, 0, 0}, {5, 0, 0}, {5, 1, -356.4}, {6, 1, 9.945}, {7, 0, -7.512}};

comp = {1, 2, 6, 7, 11, 12, 16, 17};

Join @@ Values @ KeyTake[comp] @ GroupBy[First] @ list


{{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}}

Pick[mymatrix, Times @@ BitXor[compvector, #] & /@ mymatrix[[All, 1]], 0]

(*{{1, 1, -56}, {1, 2, 3.06}, {2, 0, -30.02}, {6, 1, 9.945}, {7, 0, -7.512}}*)

• I like this method and have used it myself but in this case I don't see the advantage over simpler options. This is an order of magnitude slower than simply using Alternatives and more than two orders slower than using a Dispatch (hash) table, on the first of the two tests in my answer. Aug 10, 2013 at 6:55
• @Mr.Wizard You are right, I haven't run performance test. Aug 10, 2013 at 7:18