TemplateEvaluate, TemplateUnevaluated, and TemplateVerbatim [Undocumented] - What do you know?

Question

Let's discuss some undocumented Template-related functionality. Specifically, what do you know about TemplateEvaluate, TemplateUnevaluated, and TemplateVerbatim? Do you have any preferred "design patterns" that provide enhanced evaluation control for templates?

In a previous post (https://mathematica.stackexchange.com/a/239810/76328 -- RFC on ReplaceThen and ReplaceAt), I dabbled with using Mathematica's Template capabilities for the first serious time, but found that I had to use some undocumented functionality to provide the level of evaluation control I desired.

The main function in that class was TemplateEvaluate, which doesn't have its own documentation page, but is shown once in an example provided in the TemplateApply documentation. From that documentation:

TemplateApply can be used to build complex expressions:

test = TemplateApply[
  Hold[
   VerificationTest[
    TemplateSlot["test"],
    TemplateEvaluate[TemplateSlot["test"]]
    ]
   ], 
  <|"test" :> 4 + 4|>
  ]

I also dabbled with two other functions: TemplateUnevaluated (didn't need this in the end) and TemplateVerbatim. I found these by performing a search for "template" using SearchSystemSymbols below:

With[{symbolsSystem = Flatten@{Names@"System`*", Names@"Internal`*"}},
  SearchSystemSymbols = 
   Function[searchFor, 
    Cases[symbolsSystem, _?(StringContainsQ[searchFor, 
        IgnoreCase -> True])], Listable];]

I couldn't find a single mention of the latter functions on the internet, StackOverflow or otherwise. But: I suspect some of you may know about them and/or use them, so let's share what we know about them. One of the main components I judge any language by is its metaprogramming capabilities, and evaluation control is a fairly fundamental aspect of metaprogramming.

I'll share what I know so far about these functions in an answer to my own question.

Answer 1

To start the discussion on my own question... My basic understanding (or "what I think I understand") of these functions so far is as follows...

---

Function 1: TemplateEvaluate:

• Causes the contents of its argument to be evaluated at the time TemplateApply is invoked, whether or not those contents are wrapped inside some function that has HoldFirst, HoldRest, HoldAll, or HoldAllComplete attributes.
• Note that Template-related functions within TemplateEvaluate are not "evaluated first", so e.g. if you perform TemplateApply[TemplateEvaluate@Print@TemplateSlot@1, "test"], you will literally print TemplateSlot@1 -- so you often will need to further control the evaluation order by including Unevaluated after TemplateEvaluate. For example, TemplateApply[ TemplateEvaluate@Unevaluated@Print@TemplateSlot@1, "test"] will print "test" as-expected.
• Note that use of TemplateSlot within TemplateEvaluate will not evaluate to the verbatim expression contained within that TemplateSlot. For example, the following will not print "1+1":

TemplateApply[
 TemplateWith[{"test" :> TemplateExpression[1 + 1]},
  TemplateEvaluate@Unevaluated@Print@TemplateSlot@"test"
  ]
 ]

Rather, it will print: Templating`Evaluator`PackagePrivate`apply[Inherited,1+1].

If you actually want to transform one TemplateExpression into another TemplateExpression at the time TemplateApply is invoked, one way to do that is via the following:

TemplateSubstitute = 
  TemplateApply@*TemplateEvaluate@*
   TemplateVerbatim[TemplateExpression]@*TemplateExpression@*
   TemplateSlot;

TemplateApply[
 TemplateWith[{"test" :> TemplateExpression[1 + 1]},
  TemplateEvaluate@Unevaluated@Print@TemplateSubstitute@"test"
  ]
 ]

This will print TemplateExpression[1+1] as-expected. It may be a bit bloated, but IMO, Templates should never be used at runtime anyways, but rather only for "factory functions", so it doesn't really matter.

---

Function 2: TemplateUnevaluated:

This function appears to "stop the evaluation chain of TemplateEvaluate", and otherwise, vanish when TemplateApply is invoked. For example:

TemplateApply[
 Hold@TemplateEvaluate[a + a + TemplateUnevaluated[b + b]]
 ]

Prints: Hold[2 a + (b + b)]

I found this behavior by spelunk'ing through the code for TemplateApply itself, which I will cover below.

---

Function 3: TemplateVerbatim:

This function appears to cause the contents of its argument to be unaffected by a single invocation of TemplateApply, then it vanishes. For example:

TemplateApply@TemplateVerbatim@TemplateExpression[1 + 1]

Prints: TemplateExpression[1+1], while:

TemplateApply@TemplateApply@TemplateVerbatim@TemplateExpression[1 + 1]

Prints: 2

Note that if you refer to any TemplateSlots in an expression wrapped in TemplateVerbatim, those slots need to be available at the time that an invocation of TemplateApply actually transforms the Template-related expression into a normal expression for the argument to TemplateVerbatim. For example, the following:

TemplateWith[
   {"test" :> "test"},
   TemplateVerbatim@TemplateSlot@"test"
   ] // TemplateApply // TemplateApply

Will print: Missing[SlotAbsent,test]. A pattern I found useful (discussed earlier -- refer to "TemplateSubstitute" above) is transforming one TemplateExpression into another TemplateExpression. That is useful to do when building-up larger expressions via smaller sub-expressions defined in a TemplateWith. To do that, I use this type of pattern:

TemplateWith[
  {"test" :> "test"},
  TemplateEvaluate@
   TemplateApply@
    TemplateVerbatim[TemplateExpression]@TemplateSlot@"test"
  ] // TemplateApply

Note that the TemplateEvaluate is required here -- you will get "Missing[SlotAbsent,test]" without it. The above is an overly-trivial use-case; a more illustrative example that shows why you might want to use this pattern, using my earlier definition of TemplateSubstitute is:

TemplateApply[
 TemplateWith[
  {"test" :> TemplateExpression[1 + 1]}, 
  Hold@TemplateEvaluate[Times @@@ TemplateSubstitute@"test"]]
 ]

The output of the above is: Hold[1 * 1]. Note that the result of TemplateSubstitute itself has a head of TemplateExpression, hence the reason for using @@@ versus @@ in the above example.

---

The above were really the only undocumented Template-related functions I ran into, but nonetheless, IMO, they are some of the main functions which support Template evaluation control, which make templates significantly more useful for me.

There is probably more to some of these functions than I've learned by experimentation -- TemplateVerbatim and TemplateUnevaluated have some interesting code, which I don't claim to fully follow.

If you've read this far through my post, thanks, and let's start a discussion and share some design patterns showing how Mathematica's Template functionality is useful. If you're particularly interested in this subject, you can continue reading further, where I'll describe how I figured out what little I know so far about the noted functions.

---

I noted earlier that I found most of what I've learned by spelunk'ing through the code for TemplateApply itself. As an aid, I made a handy cheat-sheet that can be used to help decipher the behavior of these functions.

I won't post the actual Mathematica code here on StackOverflow, because I'm not sure if it's considered "proprietary", but I'll post the code to generate the cheat-sheet on your own copy of Mathematica. First, MUCH thanks to @LeonidShifrin for developing and sharing his CodeFormatter library, which I used. I only used the lower-level FullCodeFormat function (which converts boxes to beautified-boxes), because I wanted to do my own Grid-style formatting.

In the event that the author reads this post, I have one suggestion for CodeFormatter: adding a simple Format[#, InputForm] will support listing any functions which contain things like Dispatch, which are "nominally but not really" atomic. You might consider doing this either always or conditionally.

To generate your own cheat-sheet, run the following:

Import["https://raw.github.com/lshifr/CodeFormatter/master/\
CodeFormatter.m"]

templateSymbolsPackageScope = Thread@HoldForm@{
     Templating`PackageScope`eval,
     Templating`PackageScope`$InheritedValues,
 Templating`PackageScope`$NullValues,
     Templating`PackageScope`$TemplateEvaluate,
 Templating`PackageScope`$TemplatePattern};
templateSymbolsPackagePrivate = Thread@HoldForm@{
     Templating`Evaluator`PackagePrivate`replaceTemplate,
     Templating`Evaluator`PackagePrivate`replaceTemplateNoInheritance,
     Templating`Evaluator`PackagePrivate`replaceTemplatePure,
     Templating`Evaluator`PackagePrivate`insert,
     Templating`Evaluator`PackagePrivate`templateApply,
     Templating`Evaluator`PackagePrivate`activateTemplate,
     Templating`Evaluator`PackagePrivate`apply,
     Templating`Evaluator`PackagePrivate`$ActivationReplacements
     };

ClearAll[NoContext]
SetAttributes[NoContext, Listable]
NoContext[HoldForm[symbol_]] := SymbolName@Unevaluated@symbol

ClearAll[GetDefinition, SymbolHold]
SetAttributes[GetDefinition,(*HoldAll*)Listable]
SetAttributes[SymbolHold, HoldFirst]

GetDefinition[HoldForm[symbol_] | symbol_Symbol] :=
 
 With[{attributes = Attributes@Unevaluated@symbol},
  Unprotect[symbol];
  ClearAttributes[symbol, ReadProtected];
  With[
   {definition = 
     MakeExpression@
        StringReplace["\n \n" -> ","]@
         Function["{" <> # <> "}"]@ToString@Format[#, InputForm] &@
      Definition@symbol},
   Attributes[symbol] = attributes;
   
   Column[#, Frame -> All] &@
      Map[RawBoxes@*FullCodeFormat@*
         ToBoxes]@(Replace[#, SymbolHold[s_] :> s, {3, Infinity}, 
           Heads -> True] &)@(Replace[#, 
            s_Symbol :> 
             RuleCondition@
              Apply[SymbolHold]@
               MakeExpression@SymbolName@Unevaluated@s, {3, Infinity},
             Heads -> True] &)@ReleaseHold@Map[HoldForm, #, {2}] &@
    definition
   ]
  ]

SymbolsWithDefinitions[symbols_, header_ : Nothing] := 
 Prepend[header]@Transpose@{NoContext@symbols, GetDefinition@symbols}
Hdr[contents_] := {Style[contents, Bold, Red], SpanFromLeft}

Grid[#, Frame -> All] &@Catenate@{
   SymbolsWithDefinitions[templateSymbolsPackageScope, 
    Hdr@"Templating`PackageScope`"],
   SymbolsWithDefinitions[templateSymbolsPackagePrivate, 
    Hdr@"Templating`Evaluator`PackagePrivate`"]
   }
```