# professionalize/optimize my code that calls a function

I have the following construction, which defines a function that I subsequently call in a loop multiple times. I use this very often, but I have never looked into if there is a more professional way to do it in Mathematica. I would be happy to get feedback on my following MWE

g[x_] := Log[x] + 1000;

f[a_, x_, c_] := {
temp = 4*a*c + g[a] // N;
a + temp};

data = {};
For[i = 1, i < 10, i++,
data = Append[data, {i, f[i, 1, 1][[1]]}]]


EDIT: The inner part of the function f is not important, it could be anything.

-
I would like this Q&A to be called something like "the dangers of using append in a loop". – Jacob Akkerboom Dec 22 '13 at 11:25
Append copies the list that is its first argument every time it is called. About 1/2 n^2 copies of elements are made this way, where n is the length of the resulting list. You really only need of the order of n instructions to do this, for example by using a linkedList, or by figuring out the size of the of the result list beforehand. – Jacob Akkerboom Dec 22 '13 at 11:31
Not sure what you're trying to optimize? The function? Or "the code that calls the function"? – cormullion Dec 22 '13 at 11:38
– Yves Klett Dec 22 '13 at 11:39
See also this by Daniel Lichtblau. This Q&A is related, but that one still requires some work. Also related is this but the main issue there is something else. – Jacob Akkerboom Dec 22 '13 at 12:29

The function g appears to be defined correctly.

The function f uses a global Symbol without localizing it. Either use Module or write the function in such as way that this Symbol is not needed. For example:

f[a_, x_, c_] := Module[{temp},
{temp = 4*a*c + g[a] // N;
a + temp}
];


Or:

f[a_, x_, c_] := {a + N[4*a*c + g[a]]};


The accumulation of data is inefficient in both syntax and computation. Use the built in iterator constructs such as Do, Table, Sum, Fold, and Nest rather than For loops whenever possible. See Alternatives to procedural loops and iterating over lists in Mathematica.

Further, your use of Append causes reallocation of the low level array that Mathematica uses for a List, therefore your Append will take time proportional to the length of the list rather than a constant time. See Select performance and Looking for a way to insert multiple elements into multiple positions simultaneously in a list (Notes and Timings section). If you must accumulate results incrementally use Sow and Reap, or linked lists.

-
exactly what i was looking for, thanks – BillyJean Dec 22 '13 at 11:50
@BillyJean Glad I could help. :-) – Mr.Wizard Dec 22 '13 at 12:00
Personally I like not localising variables and catching unlocalised stuff later. My answer here is an example of this. I often make functions to show my intent, but inline them anyway so that they are not really used as functions. – Jacob Akkerboom Dec 22 '13 at 12:49
How does AppendTo fare in comparison to Append in terms of speed and memory efficiency? – shrx Dec 22 '13 at 23:18
@shrx AppendTo has the same performance; AppendTo[x, n] is merely a shorthand for x = Append[x, n]. Speed is not really an issue with short lists, but since I believe in writing code that scales well I still repetitive Append operations. Compare: small = Range[10]; Timing @ Do[AppendTo[small, n], {n, 1000}] to big = Range[1*^7]; Timing @ Do[AppendTo[big, n], {n, 1000}] -- huge difference. (More than 4000 times slower on my system.) – Mr.Wizard Dec 23 '13 at 8:06

You can simplify your function definitions and you can use Table rather than For loop,ie.

g[x_] := Log[x] + 1000;
f[a_, x_, c_] := a+ 4*a*c + g[a] // N;
data=Table[{j,f[j, 1, 1]}, {j, 9}]


This yields the same result as your original code.

-
thanks for your contribution, however I am interested in "general optimization/suggestions", not anything that depends on what I specifically wrote inside f, please see my edit in OP. Thanks for comment on Table – BillyJean Dec 22 '13 at 11:33
@BillyJean ok. I look forward to other answers... – ubpdqn Dec 22 '13 at 11:36

This is perhaps a bit advanced, but I feel the technique used deserves to be shown.

In your case you can use Compile, as your functions deal with numeric values only. If f and g can be anything in a broader sense, then this solution method will not always work.

I have made a slight adaptation to your functions f and g. In your case you store the integer i as the first element of a list. However i can be inferred from the position of the element of the list, so I will not store it. This will also be more convenient for use in Compile, as our result Compile can not be a mix of integers and real numbers. Furthermore, I have changed all constants that were previously integers into real numbers, especially because that works nicely in Compile.

g[x_] := Log[x] + 1000.;

f[a_, x_, c_] :=
(temp = 4.*a*c + g[a];
a + temp);


Now we Compile

cfu =
ReleaseHold@
Hold[Compile][
Hold[{{size, _Integer}}],
(
Hold[
Block[
{temp, res, iiii}
,
res = ConstantArray[0., size];
For[iiii = 1, iiii <= size, iiii++,

res[[iiii]] = f[iiii // N, 1., 1.]
];
res
]
] /. DownValues[f]
) /. DownValues[g]
,
CompilationTarget -> "C"
]


It looks a bit strange perhaps, but for me this use of DownValues is a standard way to inline function definitions. For more on this, see my answer here

Comparison

nnnn = 1*^4;


nnnn will be the size of the data we use in this section.

(res = cfu2[nnnn]) // Timing // First


0.000307

(
data = {};
For[
i = 1,
i <= nnnn,
i++,
data = Append[data, {i, f[i, 1, 1]}]]
) // Timing // First


0.444171

So the timings favor the compiled function. The results are the same

data[[All, 2]] == res


True

There is a difference in how the two are stored behind the scenes though

<< Developer

PackedArrayQ@res
PackedArrayQ@data


False
True

To get the original "data format" back, you could do

data == Transpose[{Range[nnnn], res}]


True

Comparison with ubpdqn

(data2 = Table[{j, f2[j, 1, 1]}, {j, nnnn}]; ) // Timing // First


0.050244

data2 == data


True

Reaction to comment

Possibly temporary

The technique using DownValues can also be used with uncompiled functions. Here is an example and a comparison of timings

nnnn2 = 1*^5;

ff[x_] := ff2[x];
ff2[x_] := ff3[x];
ff3[x_] := x;

(res21 = Table[ff[x], {x, nnnn2}]) // Timing // First


0.052170

Now with inlining

SetAttributes[inl, HoldAll]
inl[expr_, f_] := ReleaseHold[Hold[expr] /. DownValues[f]]

gg3[x_] := x;
inl[gg2[x_] := gg3[x], gg3];
inl[gg[x_] := gg2[x], gg2]
(res22 = inl[Table[gg[x], {x, nnnn2}], gg]) // Timing // First


0.001950

(res23 = Table[x, {x, nnnn2}]) // Timing // First


Note that the last expression is basically equivalent to evaluating Table[x, {x, nnnn2}]. The overhead to make that expression is very small.

0.001767

res21 === res22 === res23 === Range[nnnn2]
`

True

-
The inlining method you use with DownValues has a speedup purpose only within Compile or can help also in uncompiled code? I would say the former, but I am not sure. – user8074 Dec 22 '13 at 13:21
@user8074 It can help with both :). See the last section of my answer, where I reply to your comment. – Jacob Akkerboom Dec 22 '13 at 13:38