3
$\begingroup$

I am running this on a very large list, it is important to notice that some times the dates in v occure several times, and I want the selected list to show the same amount. This produces the right output but it is way to slow. How can I speed it up.

both={{{1996, 1, 5, 0, 0, 0.},2}, {{1996, 1, 8, 0, 0, 0.},3}, {{1996, 1, 11, 0, 0, \
0.},1}}
v={{1996, 1, 5, 0, 0, 0.}, {1996, 1, 8, 0, 0, 0.}, {1996, 1, 11, 0, 0, 
  0.}, {1996, 1, 11, 0, 0, 0.}, {1996, 1, 11, 0, 0, 0.}, {1996, 1, 11,
   0, 0, 0.}, {1996, 1, 11, 0, 0, 0.}, {1996, 1, 11, 0, 0, 0.}, {1996,
   1, 11, 0, 0, 0.}, {1996, 1, 12, 0, 0, 0.}, {1996, 1, 15, 0, 0, 
  0.}, {1996, 1, 16, 0, 0, 0.}}

Select[both, Function[{f}, MatchQ[f[[1]], #]], 1] & /@ v

$\endgroup$
8
$\begingroup$

Something like

v /. Dispatch[Thread[both[[All, 1]] -> both]]

should operate much faster, especially on large lists.

Surround with something like cases, e.g.,

Cases[v /. Dispatch[Thread[both[[All, 1]] -> both]], {{__}, _}]

For only "changed" to be in list.

Many, many ways to do this, btw... e.g., if the output desired is always in order of both as in your example, and you only want "matched" elements modified, this should be very quick:

MapThread[ConstantArray, {both,Tally[Join[both[[All, 1]], v]][[;; Length@both, 2]] - 1}]

Note that these output slightly different nesting than your small example, if that's an issue, let me know. You can use Flatten or Join to rearrange result less non-matches, e.g.:

Join @@ MapThread[ConstantArray, {both, 
   Tally[Join[both[[All, 1]], v]][[;; Length@both, 2]] - 1}] 
$\endgroup$
  • $\begingroup$ I see that I have to check if function goes true on the entire list and therefor it goes slow, however in this case you end up with many values that I need to delete in this answere its "deleted" by only picking the values that match the pattern I want, however there has to be a simpler and quicker way? I would like to point out that my dataset is close to 100 000 000 inputs. $\endgroup$ – ALEXANDER Mar 15 '14 at 23:00
  • $\begingroup$ Perfect thank you! $\endgroup$ – ALEXANDER Mar 15 '14 at 23:05
  • $\begingroup$ @ALEXANDER: LOL, guess I was mind-reading... glad it helped, and thanks for accept. $\endgroup$ – ciao Mar 15 '14 at 23:07
  • $\begingroup$ Haha yes you were! Add it to your CV! $\endgroup$ – ALEXANDER Mar 15 '14 at 23:09
3
$\begingroup$

In V10, you can use Association to speed this up a bit:

bothA = AssociationThread[both[[All, 1]] -> both];
Lookup[bothA, v, {}]       (* gives OP's output with {} for missing dates *)
Lookup[bothA, v, Nothing]  (* omits the {} from output *)

Large example
-- Update: forgot definition of dates, which had been lost; had to recompute results.

dates = DateList /@ DateRange[DatePlus[Today, {{-50, "Year"}}], Today, { {1, "Day"}}];
SeedRandom[0];    (* for reproducibility *)
both = Transpose[{#, RandomInteger[{1, 9}, Length@#]}] &@
   RandomSample[dates, Round[0.9 Length@dates]];
v = RandomChoice[dates, 10^7];

The use of Association is about twice as fast as @ciao's Dispatch:

(bothA = AssociationThread[both[[All, 1]] -> both];
   me2 = Lookup[bothA, v, Nothing]) // Length // RepeatedTiming
(*  {3.35, 8999973}  *)

(v /. Dispatch[Thread[both[[All, 1]] -> both]]) // Length // RepeatedTiming
(*  {6.994, 10000000}  *)

Even more if you include trimming the missing dates:

(c1 = Cases[
     v /. Dispatch[Thread[both[[All, 1]] -> both]], {{__}, _}]) // Length // RepeatedTiming
(*  {11., 8999973}  *)

And faster than @ciao's Tally method:

(c2 = Join @@ 
     MapThread[
      ConstantArray,
      {both, Tally[Join[both[[All, 1]], v]][[;; Length@both, 2]] - 1}]) // 
  Length // RepeatedTiming
(*  {12.0, 8999973}  *)

Check equivalence:

me2 == c1
Sort@me2 == Sort@c2  (* the Tally method collects them in a different order *)
(*
  True
  True
*)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.