# CompiledFunction showing its contents

CompiledFunctions usually appear very compactly, for instance:

comp = With[{d1 = {{1, 2}, {3, 4}}},
Compile[{{x, _Real, 1}},
Module[{y},
y = x^2;
d1.x + y]
]
]


However, sometimes this gets "unpacked". One obvious way is:

FullForm[comp]


However this also does it:

Hold[Evaluate[comp]] // Replace[#1, {Hold[x_] :> x}, Infinity] &


Hold[CompiledFunction[{10, 11., 5468}, {{_Real, 1}}, {{3, 1, 0}, {3, 1, 4}}, {{{{1, 2}, {3, 4}}, {2, 2, 2}}, {12, {2, 0, 1}}, {3, {2, 0, 0}}}, {0, 2, 0, 0, 5}, {{40, 56, 3, 1, 0, 3, 1, 1}, {42, "CoerceTensor", 2, 0, 0, 2, 2, 2, 3, 2, 3}, {42, "Dot", 3, 2, 3, 3, 1, 0, 2, 0, 1, 3, 1, 4}, {44, 4, 1, 4}, {1}}, Function[{x}, Module[{y\\$}, y$$= x^2; {{1, 2}, {3, 4}}.x + y$$]], Evaluate]]

Why is this? There is no Hold in the original expression for this to act on.

Furthermore, if we replace the levelspec Infinity by All (side question: what is the difference? All doesn't seem to be documented in this case), it doesn't unpack. And if we replace d1 by e.g. 3. and the Dot by multiplication, or get rid of d1 entirely, it also doesn't unpack.

Is this a bug (I'm using "11.0.1 for Mac OS X x86 (64-bit) (September 21, 2016)"), or can I find somewhere under what conditions this happens?

(The line of code above is useless ofcourse, but in reality instead of comp I have a bigger expression, that includes something like comp, that is being put together. The "unpacked" form of the CompiledFunction still works, but makes the code unreadable.)

This has to do, AFAIR, with whether the expression is marked internally as "evaluated" or perhaps "valid object." If it is so marked, it will be typeset by the Front End in the summary-box way; if it is not evaluated (or not valid), you will get the input form. This happens for other functions besides CompiledFunction, too. I don't believe there is a way at the user level to access the flag(s) that indicate the "evaluated" status of an expression. @WReach's traceView* functions seem to infer it from the evaluation sequence and marks them "inert", if you wish to explore the difference between Hold[Evaluate[comp]] and Hold[Evaluate[comp]] // Replace[#1, {}, Infinity] &.

What seems to happen is that when Replace scans the expression, it must do something that marks the expression as needing reevaluation, even if it does nothing:

Hold[Evaluate[comp]] // Replace[#1, {}, Infinity] &


In fact, in the case in question, any level specification of 5 or higher triggers the behavior:

Hold[Evaluate[comp]] // Replace[#1, {}, {5}] &


To get the summary box back, the CompiledFunction needs to be reevaluated. One way is with the following replacement:

Hold[Evaluate[comp]] // Replace[#1, {}, {5}] &
% /. cf_CompiledFunction :> Evaluate@cf /; True


Here's the same thing applied to InterpolatingFunction:

ifn = Interpolation[N@Range[4]^2];
Hold[Evaluate[ifn]] // Replace[#1, {}, Infinity] &
% /. f_InterpolatingFunction :> Evaluate@f /; True


Here are the outputs of traceView2[] on the two examples. The important difference is in the second-to-last line, which shows whether the CompiledFunction is marked "inert" or not:

• At least some of these flags may be tested with SystemPrivateNoEntryQ and SystemPrivateValidQ (there is also SystemPrivateMDataQ but I've never used it). – b3m2a1 Dec 19 '18 at 19:22
• The relevant flag is ValidQ. Just tested. – b3m2a1 Dec 20 '18 at 21:46
• @b3m2a1 How did you test it? It does not work for me....I mean it does test if the expression is, say, a valid CompiledFunction, but it does not test if it is "inert." It's the "evaluated" status that is relevant to the Q in the OP. – Michael E2 Dec 20 '18 at 22:51
• My guess is you need to use SystemPrivateHoldValidQ. See also my answer for some more details. – b3m2a1 Dec 20 '18 at 22:52
• @b3m2a1 Thanks, I got it. I needed to mimic the MakeBoxes code. (+1) -- There still seem to be distinct "valid" and "evaluated" flags; or the "valid" flag has more than two states (e.g. True, False, Evaluate). For instance foo = CompiledFunction[a, b, c] generates an error but re-executing foo does not. -- Also, while Replace unpacks the instance of d1 in the compiled code, if I unpack it directly, the expression is still valid. There do not seem to be other packed arrays in comp, none that I can find at least. – Michael E2 Dec 21 '18 at 15:31

This has everything to do with a special bit called ValidQ that is used all over the place in Mathematica. Replace by dint of its scanning and likely unpacking of some array of bytes forces a copy of the expression and thus removes this bit.

Just to check that this is the issue:

CompiledFunction // FormatValues // Keys

{HoldPattern[
MakeBoxes[ElisionsDumpcf : CompiledFunction[BoxFormargs__],
BoxFormfmt_] /;
BoxFormUseIcons &&
SystemPrivateValidQ[Unevaluated[ElisionsDumpcf]]]}


We can see it is indeed checking for ValidQ.

Here's another example:

Extract[
Replace[Hold[Evaluate[comp]], {Hold[x_] :> x}, Infinity],
1,
SystemPrivateHoldValidQ
]

False


We can also use this to make invalid input behave like a CompiledFunction:

SystemPrivateHoldSetValid[
CompiledFunction["asdasdasd", "asdasdasd", "asdasdasd",
"asdasdasd", "asdasdasd", "asdasdasd", "asdasdasd", "asdasdasd"]]@
1;

CompiledFunction::cfct: Number of arguments 1 does not match the length 8314036833820636001 of the argument template.


It actually thinks that's a CompiledFunction because of this bit. It's powerful.

This is also one of those sneaky places where the object-oriented nature of Mathematica's implementation sneaks through, too:

Replace[Hold[Evaluate[comp]], {}, Infinity] === Hold[Evaluate@comp]

True


But one isn't ValidQ and the other is.

Replace[x, ..., All] is documented. See section "Details and Options" in the documentation of Replace.

There we read that Infinity is equivalent to level spec {1, Infinity} so that the head of the expression is ignored. So in your example, just no replacement is happening.

However, All is equivalent to level spec {0, Infinity}, taking care also if the head of the expression.

But tell me: Why do you not use ReplaceAll with the very nice postfix syntax /. in the first place?

Hold[Evaluate[comp]] /. Hold[x_] :> x

• Ah of course, I read that before but forgot about it. For some reason it's not documented for Cases, where the use is the same. But although using ReplaceAll instead (I don't think I had a particular reason not to) avoids the unpacking, I would like to know what's going on with that. – Jansen Dec 19 '18 at 15:21
• @Jansen ReplaceAll never gets below depth 0` and hence the internal expression never gets invalidated and remains in its original state and hence retains its bits marking it as valid. – b3m2a1 Dec 20 '18 at 21:33