# Functional programming Style & Performance

Hello i have been spending time to convert my R tools into Mathematica Packages mainly because i like the functional programming style in Mathematica. In doing so it seems that i run into one conundrum. On one had functional programming suggests to avoid intermediate variables on the other side, a simple example like below clearly shows the potentially significant performance difference when avoiding intermediate (pre-calculated) steps.

do i miss any better solution, or is this really a case where functional style, just looses out ?

• dfm is simply a List of Lists with some syntactic sugar
• dfm["SER"] is a vector of 25k real numbers
• dfm["serial"] is a vector of 25k string factors

(* check out the add variation as function of model *)
uSerial  = DeleteDuplicates[dfm["serial"]];
tt       = Transpose@{dfm["SER"], dfm["serial"]};
foo[z_] := Select[tt, (Last[#] == z) &][[All, 1]];
serMhdd = Mean /@ Map[foo[#] &, dfm["serial"]] // N; // AbsoluteTiming

uSerial = DeleteDuplicates[dfm["serial"]];
foo[z_] := Select[Transpose@{dfm["detSER"],
dfm["serial"]}, (Last[#] == z) &][[All, 1]];
serMhdd = Mean /@ Map[foo[#] &, dfm["serial"]] // N; // AbsoluteTiming


happy holidays, Bernd

-
As a side note, serMhdd = Mean /@ Map[Pick[dfm["SER"], dfm["serial"], #] &, dfm["serial"]] // N will be much faster than using Select in this case. (I also think N is redundant if dfm["SER"] consists of Real numbers.) – Michael E2 Dec 23 '13 at 16:44
Thx a lot Michael, that's a good point. i did overlook the use of Pick with the additional selection list. it obviously makes a much larger difference here. regarding my use of //N. it was a q&d fix for the fact that the data i am working with are highly unstructured. Some are Integer some are Real, some are not even numbers. Of cause its true that in an ideal case i would have preprocessed them all into one standard form to avoid such messy syntax . anyhow thx a lot & happy holidays ;b) – bernddude Dec 24 '13 at 16:59

This is not a question of functional style per se. Functional style is, AFAICT, largely about avoiding mutable state (where possible), using functions as fundamental building blocks, using certain abstractions such as higher-order functions, closures, function composition, etc., and avoiding mixing state and behavior. I think that the value of pre computation (caching) is appreciated in FP no less than in imperative programming, even if the tools used to carry this out may be somewhat different.

In your first example, you perform the transposition only once, and store the result in tt. The usage is still bad, because your foo there depends on this global variable. But at least, you compute the value of Transpose[...] only once, and then use that already computed value. In your second example, you perform the transposition every time you apply the foo function, and this is hugely redundant.

One possible solution here is to create a closure like

With[{tt = Transpose@{dfm["detSER"], dfm["serial"]},
foo[z_] := Select[tt, (Last[#] == z) &][[All, 1]]
]


or something similar. The reason why With is needed here is that Function does not evaluate its body until it is called, while in this case you need to "inject" some preprocessed part. In this way, you stay within the FP world, but do the relevant precomputation to avoid the redundancy.

Given that your function foo is then used in Select, you could as well have it as a pure function, like e.g.

With[{tt = Transpose@{dfm["detSER"], dfm["serial"]}},
With[{foo = Function[z,Select[tt, (Last[#] == z) &][[All, 1]]]},
Mean /@ Map[foo, dfm["serial"]] // N
]
] // AbsoluteTiming


which would arguably be an even more functional solution, and here you avoid the need to introduce the intermediate variable foo.

-
Thx for the advise Leonid, interesting point, the use of with does at least avoid spreading those pesky side products into the Global environment. Even thought it doesn't quite make the code easier to read ? p.s. i think you missed one brace in the 1st with level. many thx once again, Bernd – bernddude Dec 24 '13 at 17:37
@user2562332 No problem, was glad to help. Re: brace - indeed, fixed now, thanks. Re: easier to read - well, you could assign Function[...] to foo used as a variable - which might be a bit easier to read at the expense of introducing intermediate variable (but at least not an intermediate function). You could also use my LetL macro to avoid nested With, this would also make the last code snippet a bit easier to read. – Leonid Shifrin Dec 24 '13 at 19:03
@user2562332 By the way, forgot to mention one thing. For the future questions you might have, it is a good idea to supply some minimal self-contained actually running code. In the case at hand, it was relatively easy to see what was going on without actually running the code, but in general this is not the case. Also, without such a self-contained running code, the number of people willing to even look at the problem will be much smaller than otherwise. – Leonid Shifrin Dec 24 '13 at 20:04
thx Leonid,good point, i understand and agree. in fact that is why my explanation was quite verbal. in this case to me it wasn't so much the specific code example, as the general pattern (which you addressed). however interesting enough i got a nice "extra gift" out of Michaels reply regarding the possibility to use Pick, which i overlooked. happy holidays b:) – bernddude Dec 25 '13 at 21:33
@bernddude Re: happy holidays - thanks, and the same to you :) – Leonid Shifrin Dec 25 '13 at 21:58