I needed to look at the changes in the values of about 4000 items from one year to the next. I have a table with identifiers and values for each year. Not all of the items are in each list. The tables contain the following rows, with more data to the right. enter image description here
So I did the following:

govnew = Select[data[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
govold = Select[dataold[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
intersection = Intersection[govnew[[All, 1]], govold[[All, 1]]];   
Map[(both[#] = 1) &, intersection];  
govnew2 = Select[govnew, both[#[[1]]] == 1 &];   
govold2 = Select[govold, both[#[[1]]] == 1 &];    

Which reduced the two lists to the intersection of the identifiers in both lists. It worked. But it reduces a list of Down Values to a dictionary. Doesn't use a Mathematica function in a secret and powerful way. Is there a more Mathematica like approach.

  • $\begingroup$ Could you please provide some dummy (or actual) data? btw, Scan is more typical and cleaner instead of your Map above. $\endgroup$
    – rm -rf
    Commented Jul 23, 2013 at 14:35
  • $\begingroup$ @rm-rf FWIW I'm using {data, dataold} = List @@ RandomInteger[12, {2, 20, 6}]; $\endgroup$
    – Mr.Wizard
    Commented Jul 23, 2013 at 14:44
  • $\begingroup$ +1 - Great use of DownValues. Nice! P.S., How's life? Time to actually get together. Best. $\endgroup$
    – Jagra
    Commented Jul 23, 2013 at 14:57
  • $\begingroup$ @rm-rf I never use scan. I'll find out what it does. Thanks for the suggestion. $\endgroup$ Commented Jul 23, 2013 at 17:23
  • $\begingroup$ Seems like a perfectly fine use of DownValues. An alternative might be to join the lists, then GatherBy[combinedlist,First] or something along those lines. Now discard all singletons. $\endgroup$ Commented Jul 23, 2013 at 22:41

3 Answers 3


This appears to be a perfectly legitimate use of DownValues. These are often used by experienced users as a hash table.

There are some ways you might improve this.

First, you could use the value True directly, and it's arguably better to Scan than to Map, but I've used the latter often enough myself as it rarely matters.

Scan[(both[#] = True) &, intersection];

govnew2 = Select[govnew, both[#[[1]]] &]

This DownValues hash table is often very fast as should not be dismissed. You can however do this more simply with Alternatives, e.g.:

pat = Alternatives @@ intersection;

govnew2 = Cases[govnew, {pat, __}]
govold2 = Cases[govold, {pat, __}]

Regarding the first part of your code, Pick can usually be faster though in this case also longer:

govnew = Pick[data[[2 ;;, {2, 3, 4, 6}]], Sign @ data[[2 ;;, 6]], 1]

A note regarding Alternatives

As mentioned above it is important to know that Alternatives is not a replacement for the DownValues. Alternatives are checked in order and therefore the time taken for a match will be proportionate to the number of alternatives. For applications involving a large number it will be much faster to use either DownValues or a Rule table optimized with Dispatch. For example:

range = Range[1*^5];
nums = RandomSample[range, 5000];
alts = Alternatives @@ nums;
Scan[(test[#] = True) &, nums];

Select[range, test] // Timing // First
Cases[range, alts]  // Timing // First



  • $\begingroup$ I see that True makes sense. Then I could just Select[govnew, both[#[[1]]] &], right? And I don't think I've ever used Alternatives. That would be closer to pure "Mathematica" code. $\endgroup$ Commented Jul 23, 2013 at 17:25
  • 1
    $\begingroup$ PS. I find myself wanting to use the word "Mathematical" to mean more like a Mathematica way of doing it, but it doesn't mean that. Any suggestions? $\endgroup$ Commented Jul 23, 2013 at 17:30
  • $\begingroup$ @rm-rf Thanks, everyone!! $\endgroup$ Commented Jul 23, 2013 at 17:32
  • $\begingroup$ Mathematicaish? (Maybe not..) $\endgroup$ Commented Jul 23, 2013 at 22:43
  • 1
    $\begingroup$ @Mr.Wizard I just used Alternatives[] in another application. It's elegant and clean. It's, it's ... Mathematicalicious! $\endgroup$ Commented Jul 24, 2013 at 21:32

The example below indicates what I had in mind with a comment to use Join and GatherBy. It also shows that this is roughly comparable to using DownValues.

Set up an example:

n = 6;
l1 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l2 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l1b = Map[First, GatherBy[l1, First]];
l2b = Map[First, GatherBy[l2, First]];

Using Join and GatherBy:

 l3 = Select[GatherBy[Join[l1b, l2b], First], Length[#] == 2 &];]

(* Out[178]= {2.360000, Null} *)

Using DownValues:

govnew = l1b;
govold = l2b;
Timing[intersection = 
  Intersection[govnew[[All, 1]], govold[[All, 1]]]; 
 Scan[(both[#] = 1) &, intersection]; 
 govnew2 = Select[govnew, both[#[[1]]] == 1 &]; 
 govold2 = Select[govold, both[#[[1]]] == 1 &];]

(* Out[189]= {3.780000, Null} *)

Quick check:

Sort[l3][[All, 1, 1]] === Sort[govold2][[All, 1]] === 
 Sort[govnew2][[All, 1]]

(* Out[207]= True *)

This is another problem that used to be kind of a pain back before Mathematica 10, but is now dramatically simplified by Associations and the related functions:

CompareRows[tables:{___List}] := MapThread[SortBy, 
   {(Apply[Join]) @* Values /@ KeyIntersection[GroupBy[First] /@ 
       tables], PositionIndex /@ tables}]; 

The SortBy/PositionIndex combination makes sure the rows come out in their original order. Then it's an easy matter to reproduce your results:

{govold2, govnew2} = CompareRows[{data, dataold}];

Even better, this new version works with any number of data sets for comparison, and doing so required zero extra effort on my part.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.