Extracting rows from dataset matrix

Question

I am generating a large dataset and would like to manipulate it afterwards. Here is an example matrix of my dataset. The first 3 columns are data and the last two columns are indices of the runs (each row contains the results of a single run).

mat = {{0.2, 0.1, 0.01, 1, 1}, {0.1, -0.5, 0.4, 1, 
2}, {0.4, -1.1, -0.1, 2, 1}, {0.7, 1., 1., 2, 2}, {0.3, 0.5, 0.6, 
3, 1}, {0.9, 1.1, 0.6, 3, 2}}; 
%//MatrixForm

I would like to:

a. extract row 3,4, and 5, based on the criteria that any element in the columns 1-3 is either < -1 or >1, or

b. extract rows based on the indices, e.g. row with 1,2 and row with 3,1 in the last two columns.

In either case, the result would be two new matrices, one with the remaining rows (not extracted) and another with the extracted rows.

The best I've been able to do is extract rows based on any element using Select[] and MemberQ[] which is not precise enough e.g. Select[mat, MemberQ[#, 3] &]

Many thanks.

Answer 1

The basic framework for your question is relatively simple. First you need to select items from a list based on multiple criteria, and then you need to calculate which items from your list were not included. In this case, the matrix formatting is irrelevant.

Your original code is simply a list of lists and then displayed a matrix:

mat = {{0.2, 0.1, 0.01, 1, 1}, {0.1, -0.5, 0.4, 1, 
    2}, {0.4, -1.1, -0.1, 2, 1}, {0.7, 1., 1., 2, 2}, {0.3, 0.5, 0.6, 
    3, 1}, {0.9, 1.1, 0.6, 3, 2}};
mat // MatrixForm

The key step is selecting the rows that meet your criteria. Using Select you can link multiple criteria for each row by using || to indicate "OR". (I am using Abs to streamline <-1 or >1).

criteriamet = 
  Select[mat, Abs[#[[1]]] > 1 || Abs[#[[2]]] > 1 || Abs[#[[3]]] > 1 &];
criteriamet // MatrixForm

To put the remaining rows into their own matrix, you simply have to Select those who are not members of the list you just made.

criterianotmet = Select[mat, MemberQ[criteriamet, #] == False &];
criterianotmet // MatrixForm

There are certainly ways you can improve the code for clarity and efficiency, but this should get you what you are looking for.