# Syntax highlighting for your own functions

Mathematica has a useful feature that for functions and special constructs getting passed local variables (for example Minimize or Block), the affected variables are shown in a different colour. Now when writing your own functions of this type (like the NOptimize function found in this answer), Mathematica of course doesn't know the meaning of the arguments and doesn't change the colours accordingly.

Is there a way to tell Mathematica that an argument is a set of variables to affect other arguments?

To make this post self-contained, here a small (admittedly silly) example:

SetAttributes[SequenceVars,HoldAll]
SequenceVars[vars_List,expr_]:=Block[vars,vars=Range@Length@vars;expr]
(* add syntax highlighting somehow *)

SequenceVars[{a,b,c},foo[a,b,c,d]]
(* here a, b and c, but not d, should appear in a different colour while typing *)

-

What you want is SyntaxInformation. With this, you can use every highlighting which already exists for things like Table, Solve, etc. for your own functions. You can specify the pattern of the arguments. With this you get the typical red commas if you use too many parameters. Or you can highlight locally used variables inside the function-arguments:

SetAttributes[SequenceVars, HoldAll]
SyntaxInformation[SequenceVars] = {"ArgumentsPattern" -> {_, _},
"LocalVariables" -> {"Solve", {1, 1}}};
SequenceVars[vars_List, expr_] :=
Block[vars, vars = Range@Length@vars; expr]


It looks like this in the front-end:

The usage of the "LocalVariables" highlighting is as follows. First, you choose the general type of highlighting. For instance "Solve" provides simple x or lists like {a,b} while "Plot" provides highlighting for the first element in e.g. {a,1,2}.

You can use the following settings as templates

{"Table", "Solve", "Integrate", "Limit", "Plot", "Manipulate"}


Additionally, you need to specify at which places of your function call the local variable specifications can appear. Say you want a function where in the first argument is some expression and then can follow arbitrary many lists of local variables. The definition for that would look like

SyntaxInformation[f] = {"ArgumentPattern" -> {_, __},
"LocalVariables" -> {"Solve", {2, Infinity}}}


This looks then like

and should make clear how "LocalVariables" has to be used.

## Update

Since this function seems to be of unexpected interest, I should add one more important thing: SyntaxInformation can highlight wrong options. If you use OptionsPattern in your "ArgumentsPattern" option, all non-existent options will be highlighted in red

Options[f] = {a -> 1, b -> True};
f[x_] = Identity;
SyntaxInformation[f] = {"ArgumentsPattern" -> {_, OptionsPattern[]}};


Now using f with right/wrong options gives

-
Interesting! I had no idea that SyntaxInformation can handle local variables. Big +1. – Leonid Shifrin Feb 7 '12 at 12:34
Works great except for one small detail: At least in 8.0, it's "ArgumentsPattern", not "ArgumentPattern". – celtschk Feb 7 '12 at 13:09
Wow, 16 upvotes in 2 hours, I didn't know this information was scarce since it's in the docs. @halirutan, do you know if it can be used to simulate a With/Block/Module-like local with the equal sign lala[{x=8}, ...]? – Rojolalalalalalalalalalalalala Feb 7 '12 at 18:47
@Rojo none of the given option-values for "LocalVariables" seems to fit for that. I would say it cannot be possible regarding the power of the highlighting in Block/Module/With. Consider the following two examples Table[Table[i, {i, 5}], {i, 5}] and Block[{i = 3}, Block[{i = 3}, i]]. There happens more parsing in Block/Module constructs than simple highlighting would be able to. – halirutan Feb 8 '12 at 19:52
@Mr.Wizard: Sorry for the late reply; I only noticed your question just now. Yes, it fails with "ArgumentPattern" in Version 8 (at least in 8.0.0.0), but works with "ArgumentsPattern". – celtschk Apr 10 '12 at 16:35

I don't know if this is possible for the Input - style cells, but it is certainly possible for the Program-style cells with the syntax highlighter generator I exposed in a recent answer. The work in that direction is underway, but I don't have a complete package at the moment. In fact, apart from code highlighting for languages like C, Java, etc, a strong motivation for that project for me was to create an extensible syntax highlighter for Mathematica based on it. I hope to have more to say on this soon.

Note that, in general, an implementation of the syntax highlighting of the type you mentioned requires context / semantic analysis of the program - lexical analysis is not enough.

-
A nice package, but since it didn't exactly answer my question, I've given an upvote on the linked answer instead. – celtschk Feb 7 '12 at 13:20
@celtschk I think you made the right choice. If I knew that SyntaxInformation handles local variables, I would not bother suggesting this. I expect however that the highlighter I may make based on my package will be more flexible at the end, since I have full control over the way tokens are highlighted. Even now, it does paren-matching, braces and bracket - matching, and highlights them in different colors. I don't know how easy it is to enable this for Input cells, and how flexible it is. So, I will still keep this answer. – Leonid Shifrin Feb 7 '12 at 13:26
@LeonidShifrin: Will your syntax highlighter be usable as a replacement for the built-in syntax highlighter from Wolfram for Input cells or is it restricted just to formatting/highlighting other programming languages in non-Input style cells? Also what is the current status of the project? – StackExchanger May 19 '12 at 22:15
@StackExchanger Yes, I plan to cover Mathematica as well. Potentially it will allow more than the built-in syntax highlighter. The current status of the project is that it is perfectly usable for someone to generate highlighters for any given language. In the attached notebook, I gave an example for C. I also did it for Java. It really takes just a few lines of code to write a spec for a given language, and a package for that language is generated automatically. The project needs some work though - I don't support comments yet, need to define this through styles, etc. Hope to address this soon – Leonid Shifrin May 20 '12 at 12:56
Leonid's syntax highlighter is incredible. I use it for Java, Scala, C#, Ocaml, and F# code. Just reading the code teaches you a lot about M. – Andreas Lauschke Dec 17 '12 at 19:54

This answer is focussed at more experienced users, to provide a way for them to find out more information. I do not discuss how anything works.

Information in this answer corresponds to version 10.3

Despite the fact that PrintDefinitions@SyntaxInformation gives nothing, we can still see how the function works, by doing

<< GeneralUtilites
PrintDefinitions@SystemUtilitiesGetSystemSyntaxInformation


SystemUtilitiesGetSystemSyntaxInformation is pretty much equivalent to SyntaxInformation.

From the definition of SystemUtilitiesGetSystemSyntaxInformation, we see that the full list of "options of SyntaxInformation" is

(*sio is short for SyntaxInformation options*)
sio = {"ArgumentsPattern", "OptionNames", "LocalVariables", "ColorEqualSigns"}


Of these options "OptionNames" is undocumented (see this answer for an example of use, credit to jkuczm)

The following procedure gives a list of basic template names

informationFile =
ToFileName[{$InstallationDirectory, "SystemFiles", "Kernel", "TextResources",$Language}, "FunctionInformation.m"];
(*sife is short for System Information File Expression*)
sife = If[FileType[informationFile] === File,
Get[informationFile], {}];
templatesWithNames=
Select[
sife[[1,2]]
,
Length@#>3&&#[[4]]=!= None&
][[All,{1,4}]];
DeleteDuplicates@templatesWithNames[[All,2,1]]

{"Manipulate", "Solve", "Plot", "Table", "D", "Integrate", "Limit",
"SumSign", "IntegralSign"}


Of these, {"D", "IntegralSign", "SumSign"} do not appear in the docs.

I had forgotten about the "Lexical" modifier, but that is not undocumented, see the details section of the docs). Here are some examples of templates that involve it and also one that does not.

manipulateTemplates =
Select[templatesWithNames, #[[2, 1]] == "Manipulate" &]

{"Animate",{"Manipulate",{2,∞},"Lexical"}}
{"ControllerManipulate",{"Manipulate",{2,∞},"Lexical"}}
{"Manipulate",{"Manipulate",{2,∞},"Lexical"}}
{"RepeatingElement",{"Manipulate",{2}}}


Also this is weird

"ArgumentsPattern" /. SyntaxInformation@EmbedCode

{_, Optional["/Volumes/Jenkins/workspace/Documentation.Usage.English.release/scratch"],
_., OptionsPattern[]}
`
-