# How to use Compile to efficiently apply a function to every element of a list

Let's say I have a list of random numbers;

list = Table[Random[], {100000}];

I want to apply a function f in every element of the list and take their sum. An obvious solution to that would be

Total[f[list]]

For the sake of clarity I am using here the following simple f:

f=#^2&;SetAttributes[f, Listable]

(Of course, the function Power is Listable by default

Do[Total[f[list]],{100}]//AbsoluteTiming

outputs 0.158 on my machine.)

But as always I want the fastest routine I can get. So I used Compile to reduce the running time:

fc=Compile[{{list,_Real,1}},Total[f[list]]];

Testing with

Do[fc[list],{100}]//AbsoluteTiming

I got a worse result, namely 0.288! :-(

I tested again putting f by hand inside the compiled function:

fcfast=Compile[{{list,_Real,1}},Total[list^2]];
Do[fcfast[list], {100}] // AbsoluteTiming

Output was 0.054 and I was pleased!

Why is this happening? How can I speed up my routines without having to put the function explicitly inside Compile?

Sjoerd C. de Vries answered on the question Using Apply inside Compile tracing with Needs["CompiledFunctionTools"] and the function CompilePrint that explains the timing differences but I still don't know how to improve that.

Results after applying the best solution :

I am adding here my test results using for various implementations. I added CompilationOptions -> {"InlineExternalDefinitions" -> True} which answered my question.

The list i used contained 1000000 random numbers , f=#^2& and the compiler used was the MinGW.

• @Sjoerd C. de Vries thank you for the edit. I re-edited to add a relevant question you had answered. – tchronis Nov 3 '13 at 19:03
• Well, thanks for referring to my answer. However, I have some difficulties seeing the connection. It's about Goto and Compile and does not deal with timing. – Sjoerd C. de Vries Nov 3 '13 at 19:15
• When running CompilePrint for the two compiled functions i got differences in one line. fc gave T(R1)1 = MainEvaluate[ Hold[f][ T(R1)0]] and fcfast gave T(R1)1 = Square[ T(R1)0]. – tchronis Nov 3 '13 at 19:23
• Related – Simon Woods Nov 4 '13 at 9:12
• it doesn't make a difference to the suggested solution, but I think it should be noted that f=#^2&;SetAttributes[f, Listable] does not achieve what you expect, the attributes of f don't influence how the pure function is evaluated. You woul instead want f=Function[Null,#^2,Listable]. Of course, as you mentioned, here it doesn't make a difference since Power is listable anyway. You can see the difference when using any function that isn't listable instead of Power... – Albert Retey Nov 4 '13 at 11:48

Why is this happening? How can I speed up my routines without having to put the function explicitly inside Compile?

It is happening because Compile has the attribute HoldAll

Attributes[Compile]

(* {HoldAll, Protected} *)

This means, that no evaluation of the arguments will happen. In your case the arguments to your Compile call are {{list,_Real,1}} and Total[f[list]]]. While you think that your definition of f is put inside there, exactly this does not happen. It will stay unevaluated as you can see in the output of CompilePrint

<< CompiledFunctionTools`
CompilePrint[Compile[{{list, _Real, 1}}, Total[f[list]]]]

(*.....
1   T(R1)1 = MainEvaluate[ Hold[f][ T(R1)0]]
2   R0 = TotalAll[ T(R1)1, I0]]
3   Return
" *)

Therefore, your compiled function cannot calculate the Total directly, it has to ask the Kernel every time the function is called. This results directly in a slowdown. A really easy solution to your problem is to use With because it replaces every occurrence of a named value with it's value. I didn't write variable on purpose here, because in With you don't define variables, instead it is more like constant defines. Although looking a bit awkward, this

fc = With[{f = f},
Compile[{{list, _Real, 1}}, Total[f[list]]]
]

is one solution of your problem. What happens is that the named identifier f on the left hand side of the = in the With call is connected to the value of your global f. Inside the body of the With, every occurrence of f is replaced by this value, just like you would copy and paste it manually.

Another way for injection would be

fc = Function[func, Compile[{{list, _Real, 1}}, Total[func[list]]]][f]

which has the short form

fc = Compile[{{list, _Real, 1}}, Total[#[list]]] &[f]

Or you can even use Replace in combination with Hold.

In the end, you should follow Sjoerd's suggestion and use CompilePrint to check whether you compiled function works how you expect it.

## Update:

Simon reminded me that there is another simple solution, which is the "InlineExternalDefinitions" options:

Compile[{{list, _Real, 1}}, Total[f[list]],
CompilationOptions -> {"InlineExternalDefinitions" -> True}]
• thank you very much. At last i can compile my functions the way i want. One note : New timings were much closer to that of Compile[{{list,_Real,1}},Total[list^2]]. In fact they were just x2.I used in both "With" and "&[f]" the CompilePrint and noticed that the only difference with the explicitly defined one was one extra line at the top T(R1)1 = CopyTensor[ T(R1)0]]. I have no problem with this overhead i am just wondering if it is absolutely necessary. – tchronis Nov 4 '13 at 8:02
• I think it would be good to mention the option CompilationOptions -> {"InlineExternalDefinitions" -> True} as well. – Simon Woods Nov 4 '13 at 9:13
• @SimonWoods Thanks, I forgot about this one. – halirutan Nov 4 '13 at 9:24
• Thanks @SimonWoods and Halirutan , the CompilationOptions -> {"InlineExternalDefinitions" -> True} is the most transparent and efficient of all. – tchronis Nov 4 '13 at 17:25