Selecting items in a list based on multiple tests

Question

I have an array of strings that look like this:

list = {{valueA1, valueB1, valueC1, valueD1, valueE1, valueF1, valueG1, valueH1}, {valueA2, valueB2, valueC2, valueD2, valueE2, valueF2, valueG2, valueH2}, {valueA3, valueB3, valueC3, valueD3, valueE3, valueF3, valueG3, valueH3}, ...}

I'd like to prune the list so that I generate some selectedItemsList which contains only the entries where some select set of (valueA... through valueH...) pass some test. For example, valueB... and valueH... could be real numbered values, and I'd like to only put entries in list when its the case that, for some entry k we have 1 < valueBk < 3 && -10 < valueHk... < 5 and so forth.

If list has millions of entries, so maybe we don't want to go through the list in multiple passes, what is the most efficient way to perform this selection?

Answer 1

Implementation

I will show how this can be done conveniently, based on this answer. Borrowing the function select from there:

ClearAll[select, where];
SetAttributes[where, HoldAll];
select[table : {colNames_List, rows__List}, where[condition_]] :=
  With[{selF = Apply[Function, Hold[condition] /.
      Dispatch[Thread[colNames -> Thread[Slot[Range[Length[colNames]]]]]]]},
  Select[{rows}, selF @@ # &]];

we can add a few convenience functions:

ClearAll[makeTable,getData];
makeTable[colNames_List]:= Function[data,Prepend[data,colNames]];
getData[table:{_,data___}]:= {data};

Examples

Now, here is an example. We first construct a table generator with the names of the columns being some symbols, which we find convenient to use. For example, this will generate a table with 5 columns:

tableMaker = makeTable[{valueA, valueB, valueC, valueD, valueE}];

Now, here is a small example:

testData = RandomInteger[{-10, 15}, {4, 5}]
select[tableMaker[testData], where[1 < valueB < 10 ]]

(* {{11, 9, -9, 3, -8}, {-6, 11, 11, 15, 15}, {6, -2, 11, 12, -1}, {9, -8, 8, 14, 15}} *)

(* {{11, 9, -9, 3, -8}} *)

and a large one:

testDataLrg = RandomInteger[{-10, 15}, {1000000, 5}];
getData[
   select[
       tableMaker[testDataLrg], 
       where[1 < valueB < 3 && -10 < valueE < 5 && valueC == 0 && valueA == 1 && -7 < valueD < 0]
   ]
]// AbsoluteTiming

(* {2.986258, {{1, 2, 0, -3, 1}, {1, 2, 0, -6, -7}, {1, 2, 0, -2, -7}}} *)

I'd say, the timing is Ok for a top-level Select for a table of million rows.

Notes

The main advantage of this approach is convenience: you assign symbolic names to columns, and the actual selector (pure function) is constructed automatically, freeing you from the hassle of remembering which column index means what. This is also robust w.r.t. addition of new columns, since the query is not tied to the column index. The traversal is performed only once. Also, if all your columns are numerical, it is possible to extend the above implementation by compiling the selector at run-time and increasing the performance. I can elaborate on that upon request.