6
$\begingroup$

Background (which may be skipped, unless context is needed)

I've recently begun learning Mathematica (and only up to chapter 3 of Wellin), but wanted to write some code to get a better feel for syntax before reading on. I decided a good exercise would be to obtain a list of daily prices and volumes for a stock, i.e.

P = FinancialData["QCOM", All];
V = FinancialData["QCOM", "Volume", All];

These returned, respectively,

{{{1991,12,16},0.47},{{1991,12,17},0.46},{{1991,12,18},0.48},{{1991,12,19},0.46},{{1991,12,20},0.47},{{1991,12,23},0.47},<<5447>>,{{2013,8,16},66.9},{{2013,8,19},66.33},{{2013,8,20},66.71},{{2013,8,21},66.57},{{2013,8,22},67.13},{{2013,8,23},67.15}}

and

{{{1991,12,16},143667200},{{1991,12,17},16176000},{{1991,12,18},10899200},{{1991,12,19},7299200},{{1991,12,20},5235200},{{1991,12,23},7894400},<<5448>>,{{2013,8,19},9063600},{{2013,8,20},9064900},{{2013,8,21},8471500},{{2013,8,22},6694200},{{2013,8,23},8335800}}

Probably all the dates would line up, but for sake of learning, I wanted to intersect the two lists by date. That is, obtain a list like

{{{1991,12,16},{0.47,143667200}},{{1991,12,17},{0.46,16176000}}, ... }

Problem

The canonical answer for combining lists seemed to be How do I obtain an intersection of two or more list of lists conditioned on the first element of each sub-list?, answered in main by @Mr.Wizard.

But even after an hour of investigation, I cannot seem to wrap my mind around the following:

f1[a_List, b_List] := 
   Reap[Sow[#2, #] & @@@ a ~Join~ b, a[[All, 1]] ⋂ b[[All, 1]], List][[2, All, 1]]

Here's where I've gotten:

  1. a ~Join~ b is using the infix form to join the price and volume lists.
  2. Sow[#2, #] & is a "pure function" (which confuses me, because they seem more like "anonymous" functions, distinct from "pure [virtual] functions" in C++), swapping the first and second parameters.
  3. @@@ is shorthand for Apply-at-level-1, so in other words, it's executing Sow[{1991,12,16}, 0.47]; Sow[{1991,12,17}, 0.46]; ... for all the prices, and then all the volumes. The prices (and volumes) are what's being "sowed", and each sowing is "tagged" by the date.
  4. The intersection, a[[All, 1]] ⋂ b[[All, 1]], simply returns all the "keys" (dates) that are common between the two lists.

But, beyond that, I don't understand how Reap brings it all together. Would someone explain the rest in simpler terms?

I'm a C++ developer, so I'm comfortable with programming terms. Either I'm not interpreting the documentation correctly, or my brain is fried from figuring out the rest.

Thanks very much in advance.

Improve this question
$\endgroup$
5
  • $\begingroup$ Sow throw stuff out of the window as it runs inside and Reap collects it all at the end. That is all what you need to know about these 2 functions. Reap is outside, Sow is inside. $\endgroup$
    – Nasser
    Commented Aug 25, 2013 at 5:19
  • $\begingroup$ @Nasser - I read that, and I think I get that much, but I don't understand how it's collecting it, and especially how the hell it's outputting the desired list in the end... $\endgroup$
    – Andrew Cheong
    Commented Aug 25, 2013 at 5:21
  • $\begingroup$ Also, the above solution is not working for me. (If I understood it, maybe I'd be able to debug it...) Is it possible that tagging doesn't work with dates (in list form, at least)? $\endgroup$
    – Andrew Cheong
    Commented Aug 25, 2013 at 5:33
  • $\begingroup$ Every function returns a value, and Reap returns a list containing two things: the result of the expression passed to it, followed by a list containing the results of every Sow function call inside the expression. I think you're nearly there - if you're understanding Mr.Wizard's code so soon, you're winning! :) $\endgroup$
    – cormullion
    Commented Aug 25, 2013 at 7:08
  • $\begingroup$ By the way I added a new method to my original answer that you may find both easier to understand and faster. $\endgroup$
    – Mr.Wizard
    Commented Aug 25, 2013 at 8:16

2 Answers 2

Reset to default
12
$\begingroup$

Sorry, I sometimes write rather opaque code in my own quest for brevity and a certain style.

You seem to have a pretty good understanding of my code, though I'm sorry it took as much effort as it did.

I believe in Mathematica documentation the term "pure function" is used in the manner than "anonymous function" is used elsewhere. I agree this is confusing as it seems to be confounding concepts, but maybe there is meaning in the documentation's use of this term that I haven't grasped.

Since the part of the code you have yet to understand is Reap let me give some simpler examples.

The basic use collects all values that are sown in the expression in the first argument of Reap:

Reap[Do[Sow[i^2], {i, 5}]]

{Null, {{1, 4, 9, 16, 25}}}

The result is a list with two parts:

  • The first is whatever the first argument evaluated to, here Null because that is what Do returns.
  • The second is a list of lists of sown values by tag. Here there is only one tag, the default, so there is only one list within the second part.

If items are sown to different tags they are collected in the order they first appear:

data = {{3, "a"}, {1, "b"}, {1, "c"}, {2, "d"}, {2, "e"}, {3, "f"}};

Reap[Sow[#2, #] & @@@ data]

{{"a", "b", "c", "d", "e", "f"}, {{"a", "f"}, {"b", "c"}, {"d", "e"}}}

Note that Sow returns whatever is sown but not the tag, so the first part of the result of Reap is equivalent to Last /@ data in this case. In the second part we have the elements that were sown to 3, 1, and 2 in that order. If we wish to collect only elements sown to certain tags, or tags in a certain order, we can use the second argument:

Last @ Reap[Sow[#2, #] & @@@ data, {1, 2, 3}]

{{{"b", "c"}}, {{"d", "e"}}, {{"a", "f"}}}

Finally, the third argument is a function that will receive as arguments each tag name and each list of elements collected under that tag. The default is effectively #2 &. If we instead use List we get the tags along with each result list:

Last @ Reap[Sow[#2, #] & @@@ data, {1, 2, 3}, List]

{{{1, {"b", "c"}}}, {{2, {"d", "e"}}}, {{3, {"a", "f"}}}}

The last piece of the code is [[2, All, 1]] which is shorthand for Part and simultaneously performs the work of Last as used above and First /@:

First /@ Last @ Reap[Sow[#2, #] & @@@ data, {1, 2, 3}, List]

Reap[Sow[#2, #] & @@@ data, {1, 2, 3}, List][[2, All, 1]]

{{1, {"b", "c"}}, {2, {"d", "e"}}, {3, {"a", "f"}}}

{{1, {"b", "c"}}, {2, {"d", "e"}}, {3, {"a", "f"}}}

I hope this helps.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for such an in-depth explanation. In hindsight my confusion was over Reap's parameters and the format of its returns; at the time the rest seemed to happen by magic; but your examples cleared it up wonderfully. Thanks again. $\endgroup$
    – Andrew Cheong
    Commented Aug 26, 2013 at 2:25
  • $\begingroup$ @acheong87 You're welcome. Don't forget to look at the original question again as I added new methods; I may and a fifth later if another idea pans out. $\endgroup$
    – Mr.Wizard
    Commented Aug 26, 2013 at 2:26
  • $\begingroup$ I'd like to clarify for future beginners a few things that hung me up; not because @Mr.Wizard failed to explain, but probably that these things were so basic they rightfully didn't need explanation. First, in regards to what "Sow returns", I believe the thing that's actually being returned (and being Reaped) is the return of Apply, i.e. Apply[Sow[#2, #] &, data, {1}], which is how a list of all Sow returns, is being returned. (Coming from the C++, the idea of "mapping" (as opposed to loops) is still foreign to me.) $\endgroup$
    – Andrew Cheong
    Commented Aug 26, 2013 at 2:36
  • 1
    $\begingroup$ Second, when @Mr.Wizard mentions, "The default is effectively #2 &.", what he's saying is that Reap normally returns just the sown element(s). That is, since Reap[f, expr, g] passes the tag and sown elements as parameters to g, the pure (anonymous) function #2 & would simply return the latter. (Whereas, by specifying List, Reap returns List[#, #2].) $\endgroup$
    – Andrew Cheong
    Commented Aug 26, 2013 at 2:41
  • $\begingroup$ Third, the intermediate return of Reap[Sow[#2, #] & @@@ data, {1, 2, 3}, List] (without Lasting it, and before [[2, All, 1]]) would be {{"a", "b", "c", "d", "e", "f"}, {{1, {"b", "c"}}}, {{2, {"d", "e"}}}, {{3, {"a", "f"}}}. This is where the [[2, ... selects the second list (as the first is the return list of the aforementioned Apply operation, which is useless to us), before getting [[..., All, ... of the pairs; and the ...1]] just "flattens" an extra layer of {} (I didn't see it at first, and wondered why it wasn't just the tags being returned). $\endgroup$
    – Andrew Cheong
    Commented Aug 26, 2013 at 2:46
-1
$\begingroup$

With some experimenting i found an alternative based on a transformation rule. I can read off what should happen form the code: 'transpose P and V and use a rule to name and be able to select the elements and reshuffle (transform) them to the order i want'. 

Transpose[{P,V}] /. {{{x1__}, y1_}, {{x2__}, y2_}} -> {{x1}, {y1, y2}}

Transformation Rules are powerful! You should study them if you did not already.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You're correct that transformation rules are powerful, but this doesn't appear to be performing the same operation. Also, you should be using RuleDelayed in rules such as this. $\endgroup$
    – Mr.Wizard
    Commented Aug 25, 2013 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.