Fractional Iterates
A way to obtain an approximate fractional iterate of a function is to use its Carleman matrix, which is formed from its Taylor coefficients, and then taking the appropriate $p$-th power of the matrix to obtain the series coefficients.
Note that I never said that $p$ had to be an integer; in the example given in the OP, then, we can take the square root of the Carleman matrix to get the coefficients of the semiiterate.
Using the routine CarlemanMatrix[]
from this answer, we can do this:
x0 = 0; n = 21; (* expansion point and order *)
sinCM = N[CarlemanMatrix[Sin[x], {x, x0, n}], 30];
shalfCoeffs = MatrixPower[Transpose[sinCM], 1/2, UnitVector[n + 1, 2]];
shalf[x_] = Fold[(#1 x + #2) &, 0, Reverse[shalfCoeffs]];
Notes:
- I used arbitrary precision to preserve some accuracy, as taking the square root of an unsymmetric matrix can be unstable.
- I used the "action" form of
MatrixPower[]
, as I only need the coefficients of the semiiterate, which reside in the second row of the matrix square root (hence the Transpose[]
).
- Even with the refinement in point 2, the computation takes a fair bit of time; more so if you're using a larger Carleman matrix.
To check that we now have the Taylor coefficients of the semiiterate, we can use ComposeSeries[]
:
ComposeSeries[shalf[x] + O[x]^23, shalf[x] + O[x]^23] - Sin[x] // Chop
O[x]^23
where we have exploited the fact that shalf[x]
consists only of odd powers.
One can then also consider constructing a Padé approximant from the series, like so:
shalfpade[x_] = x PadeApproximant[shalf[x]/x, {x, x0, (n - 1)/2}];
(again exploiting the oddness of the function)
Unfortunately, both the Taylor and Padé approximants have a rather limited range of validity:
Plot[{shalf[x], shalfpade[x]}, {x, -3 π/4, 3 π/4},
PlotStyle -> {RGBColor[7/19, 37/73, 22/31], RGBColor[59/67, 11/18, 1/7]}]

There may be a way to construct better approximants from the Taylor coefficients of the semiiterate, but I haven't investigated them yet.
Fractional Derivatives
We can use the general formula of the $\alpha$-th derivative of $x^k$,
$$\frac{\mathrm d^\alpha}{\mathrm dx^\alpha}x^k=\frac{\Gamma(k+1)}{\Gamma(k+1-\alpha)}x^{k-\alpha}$$
and substitute this into the Maclaurin series for the sine. This yields
sinDFrac[x_, a_] = Gamma[a + 1] Sum[(-1)^k Binomial[2 k + 1, a]
x^(2 k + 1 - a)/(2 k + 1)!, {k, 0, ∞}]
x^(1 - a) Binomial[1, a] Gamma[a + 1]
HypergeometricPFQ[{1}, {1 - a/2, 3/2 - a/2}, -x^2/4]
that is, a ${}_1 F_2$ hypergeometric function.
In the case $\alpha=\frac12$, we obtain a particularly nice form:
sinDHalf[x_] = FullSimplify[FunctionExpand[sinDFrac[x, 1/2]]]
Sqrt[2] (Cos[x] FresnelC[Sqrt[2 x/π]] + FresnelS[Sqrt[2 x/π]] Sin[x])
Plot the sine and its semiderivative:
Plot[{Sin[x], sinDHalf[x]}, {x, 0, 2 π},
PlotStyle -> {RGBColor[7/19, 37/73, 22/31], RGBColor[59/67, 11/18, 1/7]}]

f[f[x]]
? (Similar to differentiation under fourier transform) $\endgroup$NotebookOpen["http://www.internationalmathematicasymposium.org/IMS99/paper46/FractionalCalculus.nb"]
and Mikes withNotebookOpen["http://library.wolfram.com/infocenter/MathSource/7524/SemiIntegration.nb?file_id=7063"]
$\endgroup$f[f[x]]==Sin[x]
it is interesting to consider solutions with period2 Pi
. These can be conveniently expressed as Fourier series. $\endgroup$