Clever use of DownValues, or failure to understand the "Mathematica Way."

Question

I needed to look at the changes in the values of about 4000 items from one year to the next. I have a table with identifiers and values for each year. Not all of the items are in each list. The tables contain the following rows, with more data to the right.
So I did the following:

govnew = Select[data[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
govold = Select[dataold[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
intersection = Intersection[govnew[[All, 1]], govold[[All, 1]]];   
Clear[both];  
Map[(both[#] = 1) &, intersection];  
govnew2 = Select[govnew, both[#[[1]]] == 1 &];   
govold2 = Select[govold, both[#[[1]]] == 1 &];    

Which reduced the two lists to the intersection of the identifiers in both lists. It worked. But it reduces a list of Down Values to a dictionary. Doesn't use a Mathematica function in a secret and powerful way. Is there a more Mathematica like approach.

Answer 1

This appears to be a perfectly legitimate use of DownValues. These are often used by experienced users as a hash table.

There are some ways you might improve this.

First, you could use the value True directly, and it's arguably better to Scan than to Map, but I've used the latter often enough myself as it rarely matters.

Scan[(both[#] = True) &, intersection];

govnew2 = Select[govnew, both[#[[1]]] &]

This DownValues hash table is often very fast as should not be dismissed. You can however do this more simply with Alternatives, e.g.:

pat = Alternatives @@ intersection;

govnew2 = Cases[govnew, {pat, __}]
govold2 = Cases[govold, {pat, __}]

Regarding the first part of your code, Pick can usually be faster though in this case also longer:

govnew = Pick[data[[2 ;;, {2, 3, 4, 6}]], Sign @ data[[2 ;;, 6]], 1]

---

A note regarding Alternatives

As mentioned above it is important to know that Alternatives is not a replacement for the DownValues. Alternatives are checked in order and therefore the time taken for a match will be proportionate to the number of alternatives. For applications involving a large number it will be much faster to use either DownValues or a Rule table optimized with Dispatch. For example:

range = Range[1*^5];
nums = RandomSample[range, 5000];
alts = Alternatives @@ nums;
Scan[(test[#] = True) &, nums];

Select[range, test] // Timing // First
Cases[range, alts]  // Timing // First

0.049

5.289

Answer 2

The example below indicates what I had in mind with a comment to use Join and GatherBy. It also shows that this is roughly comparable to using DownValues.

Set up an example:

n = 6;
l1 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l2 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l1b = Map[First, GatherBy[l1, First]];
l2b = Map[First, GatherBy[l2, First]];

Using Join and GatherBy:

Timing[
 l3 = Select[GatherBy[Join[l1b, l2b], First], Length[#] == 2 &];]

(* Out[178]= {2.360000, Null} *)

Using DownValues:

govnew = l1b;
govold = l2b;
Clear[both];
Timing[intersection = 
  Intersection[govnew[[All, 1]], govold[[All, 1]]]; 
 Scan[(both[#] = 1) &, intersection]; 
 govnew2 = Select[govnew, both[#[[1]]] == 1 &]; 
 govold2 = Select[govold, both[#[[1]]] == 1 &];]

(* Out[189]= {3.780000, Null} *)

Quick check:

Sort[l3][[All, 1, 1]] === Sort[govold2][[All, 1]] === 
 Sort[govnew2][[All, 1]]

(* Out[207]= True *)

Answer 3

This is another problem that used to be kind of a pain back before Mathematica 10, but is now dramatically simplified by Associations and the related functions:

CompareRows[tables:{___List}] := MapThread[SortBy, 
   {(Apply[Join]) @* Values /@ KeyIntersection[GroupBy[First] /@ 
       tables], PositionIndex /@ tables}]; 

The SortBy/PositionIndex combination makes sure the rows come out in their original order. Then it's an easy matter to reproduce your results:

{govold2, govnew2} = CompareRows[{data, dataold}];

Even better, this new version works with any number of data sets for comparison, and doing so required zero extra effort on my part.