Functional programming Style & Performance

Question

Hello i have been spending time to convert my R tools into Mathematica Packages mainly because i like the functional programming style in Mathematica. In doing so it seems that i run into one conundrum. On one had functional programming suggests to avoid intermediate variables on the other side, a simple example like below clearly shows the potentially significant performance difference when avoiding intermediate (pre-calculated) steps.

do i miss any better solution, or is this really a case where functional style, just looses out ?

• dfm is simply a List of Lists with some syntactic sugar
• dfm["SER"] is a vector of 25k real numbers
• dfm["serial"] is a vector of 25k string factors

---

(* check out the add variation as function of model *)
uSerial  = DeleteDuplicates[dfm["serial"]];
tt       = Transpose@{dfm["SER"], dfm["serial"]};
foo[z_] := Select[tt, (Last[#] == z) &][[All, 1]];
serMhdd = Mean /@ Map[foo[#] &, dfm["serial"]] // N; // AbsoluteTiming

uSerial = DeleteDuplicates[dfm["serial"]];
foo[z_] := Select[Transpose@{dfm["detSER"], 
                dfm["serial"]}, (Last[#] == z) &][[All, 1]];
serMhdd = Mean /@ Map[foo[#] &, dfm["serial"]] // N; // AbsoluteTiming

happy holidays, Bernd

Answer 1

This is not a question of functional style per se. Functional style is, AFAICT, largely about avoiding mutable state (where possible), using functions as fundamental building blocks, using certain abstractions such as higher-order functions, closures, function composition, etc., and avoiding mixing state and behavior. I think that the value of pre computation (caching) is appreciated in FP no less than in imperative programming, even if the tools used to carry this out may be somewhat different.

In your first example, you perform the transposition only once, and store the result in tt. The usage is still bad, because your foo there depends on this global variable. But at least, you compute the value of Transpose[...] only once, and then use that already computed value. In your second example, you perform the transposition every time you apply the foo function, and this is hugely redundant.

One possible solution here is to create a closure like

With[{tt = Transpose@{dfm["detSER"], dfm["serial"]}, 
   foo[z_] := Select[tt, (Last[#] == z) &][[All, 1]]
] 

or something similar. The reason why With is needed here is that Function does not evaluate its body until it is called, while in this case you need to "inject" some preprocessed part. In this way, you stay within the FP world, but do the relevant precomputation to avoid the redundancy.

Given that your function foo is then used in Select, you could as well have it as a pure function, like e.g.

With[{tt = Transpose@{dfm["detSER"], dfm["serial"]}},    
  With[{foo = Function[z,Select[tt, (Last[#] == z) &][[All, 1]]]},
     Mean /@ Map[foo, dfm["serial"]] // N
  ]
] // AbsoluteTiming

which would arguably be an even more functional solution, and here you avoid the need to introduce the intermediate variable foo.