I don't have any intuition whatsoever as to what your expected output is. The error message that emerges can be dealt with if you increase your iteration and recursion limits but seeing as I don't really know exactly how BellY works I leave this to you. If you write your g function recursively, like so
der[g_, x_, n_] :=
Sum[g[k, g[0, x]] BellY[n, k, Table[g[i, x], {i, n - k + 1}]], {k, 0,
n}]
g[1, x] = x;
g[n_, x_] := g[n, x] = der[g, x, n - 1]
and evaluate:
Expand[der[g, x, 4]]
you get
x^5 + 11 x^6 + 11 x^7 + 8 x^8 + 4 x^9 + x^10 + x^11
And something more complicated, like evaluating:
Expand[der[g, x, 13]]
outputs (after some warnings re low recursion limits):
x^14 + 8178 x^15 + 1479726 x^16 + 45923846 x^17 + 459029454 x^18 +
2208858006 x^19 + 7075642044 x^20 + 16806933760 x^21 +
31816904623 x^22 + 51866622093 x^23 + 74193541415 x^24 +
98180859588 x^25 + 119082181277 x^26 + 137536650290 x^27 +
150143774552 x^28 + 157992149040 x^29 + 160050416326 x^30 +
157096816119 x^31 + 150189370525 x^32 + 139814563126 x^33 +
127454419592 x^34 + 113603426826 x^35 + 99578396788 x^36 +
85494865250 x^37 + 72430802460 x^38 + 60247338445 x^39 +
49525049687 x^40 + 40031803598 x^41 + 32046236578 x^42 +
25240371714 x^43 + 19719393876 x^44 + 15171635174 x^45 +
11589845200 x^46 + 8724377910 x^47 + 6528035076 x^48 +
4813974664 x^49 + 3532331160 x^50 + 2554783500 x^51 +
1840197183 x^52 + 1306568236 x^53 + 924434043 x^54 +
644645508 x^55 + 448470745 x^56 + 307216681 x^57 + 210203601 x^58 +
141532130 x^59 + 95259446 x^60 + 63040614 x^61 + 41765538 x^62 +
27144679 x^63 + 17701948 x^64 + 11305600 x^65 + 7250417 x^66 +
4546412 x^67 + 2871683 x^68 + 1763322 x^69 + 1096094 x^70 +
660630 x^71 + 402853 x^72 + 237206 x^73 + 142759 x^74 + 81694 x^75 +
48320 x^76 + 26994 x^77 + 15606 x^78 + 8419 x^79 + 4842 x^80 +
2476 x^81 + 1395 x^82 + 694 x^83 + 374 x^84 + 173 x^85 + 98 x^86 +
38 x^87 + 22 x^88 + 8 x^89 + 4 x^90 + x^91 + x^92
Again, with the caveat that I don't know the specific polynomial expansion so it may well be that your expression for der doesn't always converge.
derthat works for alln? – rm -rf♦ Nov 1 '12 at 15:32Tableneeds a numeric value for the iterator. Try replacing it withSumand see if it simplifies things (no guarantees) – rm -rf♦ Nov 1 '12 at 16:02