Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I needed to look at the changes in the values of about 4000 items from one year to the next. I have a table with identifiers and values for each year. Not all of the items are in each list. The tables contain the following rows, with more data to the right. enter image description here
So I did the following:

govnew = Select[data[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
govold = Select[dataold[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
intersection = Intersection[govnew[[All, 1]], govold[[All, 1]]];   
Clear[both];  
Map[(both[#] = 1) &, intersection];  
govnew2 = Select[govnew, both[#[[1]]] == 1 &];   
govold2 = Select[govold, both[#[[1]]] == 1 &];    

Which reduced the two lists to the intersection of the identifiers in both lists. It worked. But it reduces a list of Down Values to a dictionary. Doesn't use a Mathematica function in a secret and powerful way. Is there a more Mathematica like approach.

share|improve this question
    
Could you please provide some dummy (or actual) data? btw, Scan is more typical and cleaner instead of your Map above. –  rm -rf Jul 23 '13 at 14:35
    
@rm-rf FWIW I'm using {data, dataold} = List @@ RandomInteger[12, {2, 20, 6}]; –  Mr.Wizard Jul 23 '13 at 14:44
    
+1 - Great use of DownValues. Nice! P.S., How's life? Time to actually get together. Best. –  Jagra Jul 23 '13 at 14:57
    
@rm-rf I never use scan. I'll find out what it does. Thanks for the suggestion. –  George Wolfe Jul 23 '13 at 17:23
    
Seems like a perfectly fine use of DownValues. An alternative might be to join the lists, then GatherBy[combinedlist,First] or something along those lines. Now discard all singletons. –  Daniel Lichtblau Jul 23 '13 at 22:41

2 Answers 2

up vote 11 down vote accepted

This appears to be a perfectly legitimate use of DownValues. These are often used by experienced users as a hash table.

There are some ways you might improve this.

First, you could use the value True directly, and it's arguably better to Scan than to Map, but I've used the latter often enough myself as it rarely matters.

Scan[(both[#] = True) &, intersection];

govnew2 = Select[govnew, both[#[[1]]] &]

This DownValues hash table is often very fast as should not be dismissed. You can however do this more simply with Alternatives, e.g.:

pat = Alternatives @@ intersection;

govnew2 = Cases[govnew, {pat, __}]
govold2 = Cases[govold, {pat, __}]

Regarding the first part of your code, Pick can usually be faster though in this case also longer:

govnew = Pick[data[[2 ;;, {2, 3, 4, 6}]], Sign @ data[[2 ;;, 6]], 1]

A note regarding Alternatives

As mentioned above it is important to know that Alternatives is not a replacement for the DownValues. Alternatives are checked in order and therefore the time taken for a match will be proportionate to the number of alternatives. For applications involving a large number it will be much faster to use either DownValues or a Rule table optimized with Dispatch. For example:

range = Range[1*^5];
nums = RandomSample[range, 5000];
alts = Alternatives @@ nums;
Scan[(test[#] = True) &, nums];

Select[range, test] // Timing // First
Cases[range, alts]  // Timing // First

0.049

5.289

share|improve this answer
    
I see that True makes sense. Then I could just Select[govnew, both[#[[1]]] &], right? And I don't think I've ever used Alternatives. That would be closer to pure "Mathematica" code. –  George Wolfe Jul 23 '13 at 17:25
1  
PS. I find myself wanting to use the word "Mathematical" to mean more like a Mathematica way of doing it, but it doesn't mean that. Any suggestions? –  George Wolfe Jul 23 '13 at 17:30
    
@rm-rf Thanks, everyone!! –  George Wolfe Jul 23 '13 at 17:32
    
Mathematicaish? (Maybe not..) –  Daniel Lichtblau Jul 23 '13 at 22:43
1  
@Mr.Wizard I just used Alternatives[] in another application. It's elegant and clean. It's, it's ... Mathematicalicious! –  George Wolfe Jul 24 '13 at 21:32

The example below indicates what I had in mind with a comment to use Join and GatherBy. It also shows that this is roughly comparable to using DownValues.

Set up an example:

n = 6;
l1 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l2 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l1b = Map[First, GatherBy[l1, First]];
l2b = Map[First, GatherBy[l2, First]];

Using Join and GatherBy:

Timing[
 l3 = Select[GatherBy[Join[l1b, l2b], First], Length[#] == 2 &];]

(* Out[178]= {2.360000, Null} *)

Using DownValues:

govnew = l1b;
govold = l2b;
Clear[both];
Timing[intersection = 
  Intersection[govnew[[All, 1]], govold[[All, 1]]]; 
 Scan[(both[#] = 1) &, intersection]; 
 govnew2 = Select[govnew, both[#[[1]]] == 1 &]; 
 govold2 = Select[govold, both[#[[1]]] == 1 &];]

(* Out[189]= {3.780000, Null} *)

Quick check:

Sort[l3][[All, 1, 1]] === Sort[govold2][[All, 1]] === 
 Sort[govnew2][[All, 1]]

(* Out[207]= True *)
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.