Mathematica Stack Exchange is a question and answer site for users of Mathematica. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am running this on a very large list, it is important to notice that some times the dates in v occure several times, and I want the selected list to show the same amount. This produces the right output but it is way to slow. How can I speed it up.

both={{{1996, 1, 5, 0, 0, 0.},2}, {{1996, 1, 8, 0, 0, 0.},3}, {{1996, 1, 11, 0, 0, \
0.},1}}
v={{1996, 1, 5, 0, 0, 0.}, {1996, 1, 8, 0, 0, 0.}, {1996, 1, 11, 0, 0, 
  0.}, {1996, 1, 11, 0, 0, 0.}, {1996, 1, 11, 0, 0, 0.}, {1996, 1, 11,
   0, 0, 0.}, {1996, 1, 11, 0, 0, 0.}, {1996, 1, 11, 0, 0, 0.}, {1996,
   1, 11, 0, 0, 0.}, {1996, 1, 12, 0, 0, 0.}, {1996, 1, 15, 0, 0, 
  0.}, {1996, 1, 16, 0, 0, 0.}}

Select[both, Function[{f}, MatchQ[f[[1]], #]], 1] & /@ v

share|improve this question
up vote 8 down vote accepted

Something like

v /. Dispatch[Thread[both[[All, 1]] -> both]]

should operate much faster, especially on large lists.

Surround with something like cases, e.g.,

Cases[v /. Dispatch[Thread[both[[All, 1]] -> both]], {{__}, _}]

For only "changed" to be in list.

Many, many ways to do this, btw... e.g., if the output desired is always in order of both as in your example, and you only want "matched" elements modified, this should be very quick:

MapThread[ConstantArray, {both,Tally[Join[both[[All, 1]], v]][[;; Length@both, 2]] - 1}]

Note that these output slightly different nesting than your small example, if that's an issue, let me know. You can use Flatten or Join to rearrange result less non-matches, e.g.:

Join @@ MapThread[ConstantArray, {both, 
   Tally[Join[both[[All, 1]], v]][[;; Length@both, 2]] - 1}] 
share|improve this answer
    
I see that I have to check if function goes true on the entire list and therefor it goes slow, however in this case you end up with many values that I need to delete in this answere its "deleted" by only picking the values that match the pattern I want, however there has to be a simpler and quicker way? I would like to point out that my dataset is close to 100 000 000 inputs. – ALEXANDER Mar 15 '14 at 23:00
    
Perfect thank you! – ALEXANDER Mar 15 '14 at 23:05
    
@ALEXANDER: LOL, guess I was mind-reading... glad it helped, and thanks for accept. – ciao Mar 15 '14 at 23:07
    
Haha yes you were! Add it to your CV! – ALEXANDER Mar 15 '14 at 23:09

In V10, you can use Association to speed this up a bit:

bothA = AssociationThread[both[[All, 1]] -> both];
Lookup[bothA, v, {}]       (* gives OP's output with {} for missing dates *)
Lookup[bothA, v, Nothing]  (* omits the {} from output *)

Large example
-- Update: forgot definition of dates, which had been lost; had to recompute results.

dates = DateList /@ DateRange[DatePlus[Today, {{-50, "Year"}}], Today, { {1, "Day"}}];
SeedRandom[0];    (* for reproducibility *)
both = Transpose[{#, RandomInteger[{1, 9}, Length@#]}] &@
   RandomSample[dates, Round[0.9 Length@dates]];
v = RandomChoice[dates, 10^7];

The use of Association is about twice as fast as @ciao's Dispatch:

(bothA = AssociationThread[both[[All, 1]] -> both];
   me2 = Lookup[bothA, v, Nothing]) // Length // RepeatedTiming
(*  {3.35, 8999973}  *)

(v /. Dispatch[Thread[both[[All, 1]] -> both]]) // Length // RepeatedTiming
(*  {6.994, 10000000}  *)

Even more if you include trimming the missing dates:

(c1 = Cases[
     v /. Dispatch[Thread[both[[All, 1]] -> both]], {{__}, _}]) // Length // RepeatedTiming
(*  {11., 8999973}  *)

And faster than @ciao's Tally method:

(c2 = Join @@ 
     MapThread[
      ConstantArray,
      {both, Tally[Join[both[[All, 1]], v]][[;; Length@both, 2]] - 1}]) // 
  Length // RepeatedTiming
(*  {12.0, 8999973}  *)

Check equivalence:

me2 == c1
Sort@me2 == Sort@c2  (* the Tally method collects them in a different order *)
(*
  True
  True
*)
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.