Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have an array of strings that look like this:

list = {{valueA1, valueB1, valueC1, valueD1, valueE1, valueF1, valueG1, valueH1}, {valueA2, valueB2, valueC2, valueD2, valueE2, valueF2, valueG2, valueH2}, {valueA3, valueB3, valueC3, valueD3, valueE3, valueF3, valueG3, valueH3}, ...}

I'd like to prune the list so that I generate some selectedItemsList which contains only the entries where some select set of (valueA... through valueH...) pass some test. For example, valueB... and valueH... could be real numbered values, and I'd like to only put entries in list when its the case that, for some entry k we have 1 < valueBk < 3 && -10 < valueHk... < 5 and so forth.

If list has millions of entries, so maybe we don't want to go through the list in multiple passes, what is the most efficient way to perform this selection?

share|improve this question
    
Have you seen Cases or DeleteCases? This should work there: _?(-10<#<3 &). It would be better for you if you could provide a working example of those values for us to help you further –  Sosi Apr 9 at 15:48
add comment

1 Answer 1

up vote 1 down vote accepted

Implementation

I will show how this can be done conveniently, based on this answer. Borrowing the function select from there:

ClearAll[select, where];
SetAttributes[where, HoldAll];
select[table : {colNames_List, rows__List}, where[condition_]] :=
  With[{selF = Apply[Function, Hold[condition] /.
      Dispatch[Thread[colNames -> Thread[Slot[Range[Length[colNames]]]]]]]},
  Select[{rows}, selF @@ # &]];

we can add a few convenience functions:

ClearAll[makeTable,getData];
makeTable[colNames_List]:= Function[data,Prepend[data,colNames]];
getData[table:{_,data___}]:= {data};

Examples

Now, here is an example. We first construct a table generator with the names of the columns being some symbols, which we find convenient to use. For example, this will generate a table with 5 columns:

tableMaker = makeTable[{valueA, valueB, valueC, valueD, valueE}];

Now, here is a small example:

testData = RandomInteger[{-10, 15}, {4, 5}]
select[tableMaker[testData], where[1 < valueB < 10 ]]

(* {{11, 9, -9, 3, -8}, {-6, 11, 11, 15, 15}, {6, -2, 11, 12, -1}, {9, -8, 8, 14, 15}} *)

(* {{11, 9, -9, 3, -8}} *)

and a large one:

testDataLrg = RandomInteger[{-10, 15}, {1000000, 5}];
getData[
   select[
       tableMaker[testDataLrg], 
       where[1 < valueB < 3 && -10 < valueE < 5 && valueC == 0 && valueA == 1 && -7 < valueD < 0]
   ]
]// AbsoluteTiming

(* {2.986258, {{1, 2, 0, -3, 1}, {1, 2, 0, -6, -7}, {1, 2, 0, -2, -7}}} *)

I'd say, the timing is Ok for a top-level Select for a table of million rows.

Notes

The main advantage of this approach is convenience: you assign symbolic names to columns, and the actual selector (pure function) is constructed automatically, freeing you from the hassle of remembering which column index means what. This is also robust w.r.t. addition of new columns, since the query is not tied to the column index. The traversal is performed only once. Also, if all your columns are numerical, it is possible to extend the above implementation by compiling the selector at run-time and increasing the performance. I can elaborate on that upon request.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.