Finding All Operators and Their Precedence, Arity, Affix, and Associativity
Sources of Information
There are a few different sources of information about operators and their properties. I list all known sources (as of August 2018) in the table below. We will explore these sources in more detail in the remainder of this section.
- Official Documentation: The most common operators only. Usually no precedence or associativity.
WolframLanguageData[]
: Only has info for most common operators. For most operators, gives precedence rank, arity, affix. Less often associativity info is apparent.
Precedence[]
: Undocumented built-in function. When provided the long name of an operator, gives precedence. Only works with some operators and is often incorrect.
UnicodeCharacters.tr
: Gives explicit precedence, affix, and associativity for 338 operators that are a single Unicode character outside of ASCII character set. Note that this undocumented system file uses a different precedence numbering scheme than other sources and which applies only to the notebook interface.
System`Convert`MLStringDataDump`$Operators
: Undocumented list of 340 operators—characters only. Includes most operators, even single and multi-character ASCII operators. Excludes Association brackets, box operators, and obscure bracketing operators.
System`Convert`MLStringDataDump`$ExtractedOperators
: A proper subset of $Operators
above with 272 single character unicode operators. Listed for completeness.
Internal`SymbolNameQ
: Undocumented function that apparently identifies a string as a "symbol name." Computing the complement of the set of Unicode characters for which SymbolNameQ
evaluates to True gives a list of operators, including some single-character box operators found in no other list.
Note: No single source contains all operators, and all but $ExtractedOperators
and (ironically) the official documentation are necessary to obtain all available information about all operators.
The documentation and the undocumented Precedence[]
function
You will find that the table given in the Operator Input Forms documentation is sometimes incorrect and woefully incomplete. The same is true of the undocumented Precedence
function. Neither includes all of the operators, or even all precedence/precedence ranks, and neither is to be trusted.
WolframLanguageData[]
I have found that the precedence rank returned by WolframLanguageData
is more accurate than the Precedence
function. As a bonus, WolframLanguageData
gives you really nice operator synonym and FullForm data.
WolframLanguageData["Power", "PrecedenceRanks"]
{{expr1\_expr2\%expr3, Power[Subscript[expr1,expr2],expr3]}->8,
{expr1^expr2, Power[expr1,expr2]}->21,
{expr1^expr2, Power[expr1,expr2]}->21,
{expr1^expr2, Power[expr1,expr2]}->21,
{Subsuperscript[expr1, expr2, expr3],
Power[Subscript[expr1,expr2],expr3]}->21,
{expr1\^expr2\%expr3, Power[Subscript[expr1,expr3],expr2]}->21,
{\@expr\%n,Power[expr,Power[n,-1]]}->22}
Another bonus of WolframLanguageData
is that you do not already have to know the name of every function with an operator. You can ask WolframLanguageData
to just give you everything in its database, including the box sublanguage. The following will generate a list for you. Be warned that it will download data for all 5,000+ functions and ultimately give you a list of 135 (as of August 2018) {name, rankdata, shortnotation}
triples:
Select[
WolframLanguageData[
WolframLanguageData["Entities"],
{"Name", "PrecedenceRanks", "ShortNotations"}
],
#[[2]] =!= Missing["NotApplicable"] &
]
The "ShortNotations"
data provided by WolframLanguageData
leaves a lot to be desired. And, unfortunately, WolframLanguageData
will not give you every operator. However, this is the only source that gives all multicharacter box operators.
UnicodeCharacters.tr
The Mathematica system files includes a file named UnicodeCharacters.tr
. I borrow from Mr. Wizard's answer the following code, slightly modified, to read in the data from the file:
ucharTR = ReadList[
System`Dump`unicodeCharactersTR,
Word, WordSeparators -> {"\t\t"},
RecordLists -> True
];
As its name implies, the UnicodeCharacters.tr
file appears to only include characters that are single Unicode characters outside of the ASCII character set. It does not include !
or /@
, for example. Most of the records in the file are not operators. But for the operators that are included, it gives us the most detailed information of any source. Each record has up to 9 columns (fields) that we might name: (1)Hex Code, (2)Long Name, (3)Alternative Input Methods, (4)Class/Affix, (5)Precedence, (6)Associativity, (7)Left Spacing, (8)Right Spacing, (9)Rendered As. This is everything we could hope to know about the linguistics of an operator. Be aware that it uses a different precedence numbering scheme from Precedence
, so be sure not to mix up the two. Note also that the precedence in the UnicodeCharacters.tr
refers to the precedence of the frontend only. (See the third displayed example in the section "Programmatically determining properties of operators.")
We can filter out non-operators using the Class/Affix column. Let's look at the unique values in that column.
Union[Select[ucharTR, Length[#] >= 4 &][[All, 4]]]
{"Alias", "Auto", "Close", "CompoundPrefix", "Infix",
"InfixOpen", "LargeOp:Prefix", "Letter", "NonBreakingAfterLetter",
"NonBreakingBeforeLetter", "Open", "Postfix", "Prefix", "WhiteSp"}
These are the ones we want:
affixes = {"Close", "CompoundPrefix", "Infix", "InfixOpen",
"LargeOp:Prefix", "Open", "Postfix", "Prefix"};
ops = Select[ucharTR, Length[#] >= 4 && MemberQ[affixes, #[[4]]]&];
Length[ops]
337
The character 〚 is the only one listed with affix InfixOpen
, while the other matchfix (circumfix) operators are under Open
or Close
. Most of them appear to function like parentheses, but their rendering in the frontend is flakey. We also have LargeOp:Prefix
operators like ∑ (\[Sum]
), and then ∇ (\[Del]
), which is the only operator with affix CompoundPrefix
(I don't know why).
Of the sources of operators and their precedence, the UnicodeCharacters.tr
is among the largest, and since it is presumably used by Mathematica internals, you can be sure that it is also the most accurate.
Mysteriously, however, it seems to be wrong about precedence in at least one case that I've found so far: UnicodeCharacters.tr
gives CircleDot
and PermutationProduct
the same precedence, but CircleDot
does have a higher precedence than PermutationProduct
while still being lower than the next highest operator, SmallCircle
. That is to say, at least on v11.3, CircleDot
's real precedence is 606, not 605 as it's listed.
System`Convert`MLStringDataDump`$Operators
This is an undocumented list of 340 operators without any additional information. The list is only available in the notebook frontend. The undocumented function System`Convert`MLStringDataDump`OperatorQ[]
works by checking membership in this list.
This is the only list, excluding the documentation and WolframLanguageData
, that includes all single and multi-character ASCII operators. As with most other sources, no box operators are included. It also excludes Association brackets, obscure bracketing symbols, many LargeOp:Prefix operators, and a handful of other operators found in UnicodeCharacters.tr
.
The complement of Internal`SymbolNameQ
characters
Using the definition of LetterLikeQ
along with the opaque Internal`SymbolNameQ
, we can search the list of Unicode characters to find the complement of the set of characters we already know. Our search criteria is as follows:
crit[c_] := ! Internal`SymbolNameQ[c] && ! DigitQ[c] && !
StringMatchQ[c, WhitespaceCharacter] && ! MemberQ[ops, c];
Then we make a table of code/character pairs:
Select[
Table[{i, FromCharacterCode[i]}, {i, 65535}],
crit[#[[2]]]&
]
The result is a bit of a surprise: We get a seemingly random handful of ASCII characters, Association brackets, several of the LargeOp:Prefix operators also missing in other lists and a small handful of other Unicode operators, and, most interestingly, single character versions of multicharacter box operators that are not in any other source. These box operators are:
0xf7c0: \), 0xf7c1: \!, 0xf7c2: \@, 0xf7c5: \%,
0xf7c6: \^, 0xf7c7: \&, 0xf7c8: \*, 0xf7c9: \(,
0xf7ca: \_, 0xf7cb: \+, 0xf7cc: \/, 0xf7cd: \`
The list also includes the nonoperator characters 0xf767: \[ErrorIndicator]
, 0x2043: \[SkeletonIndicator]
, the ASCII delete character 0x007f
(127), the ASCII bell character 0x0007
, and 0xf3ad
, which is an unassigned noncharacter Unicode codepoint "for private use."
Other "Operators"
There are other lexemes (strings) that could be considered operators that are still missing:
- string-related characters like " and escape characters
\n
, \t
, etc.
- The character representation operators:
\[name]
, \:nn
, and \.nnnn
.
- The number representation operators:
^^
, *^
, and ``
, though the single `
character does appear in the previous subsection.
Programmatically determining properties of operators
Putting these different sources together gives you a database, but there are still some gaps in the dataset. As far as I know, though, these are the only operators anyone who isn't an employee of Wolfram knows about.
You ask specifically about whether there is "a way to get a list of all existing operators (ideally with precedence) using meta programming." We have answered the first half of the question in the affirmative, having seen how to compile a list of operators, though it took some effort to get them all.
Is there a programmatic way to fill in the gaps in the resulting dataset once you know the operators, that is, what lexical tokens make up the operators? It's easy to programmatically determine affix (prefix, infix, etc.) and arity (unary, binary, etc.) from the usage data that can be obtained from WolframLanguageData
, and we have everything we need for those operators listed in UnicodeCharacters.tr
.
Precedence is a lot harder. I have implemented a strategy that compares two operators by instantiating expressions involving the two operators and inspecting the result. The remainer of this section gives just a few examples to illustrate the fundamental reason why this strategy doesn't work.
Two synonyms of the same operator:
FullForm[Hold[a\[Conditioned]b<->c]]
FullForm[Hold[a\[Conditioned]b\[TwoWayRule]c]]
Hold[Conditioned[a,TwoWayRule[b,c]]]
Hold[TwoWayRule[Conditioned[a,b],c]]
The only binary operator that cannot be parsed like this:
a \[DirectedEdge] b \[UndirectedEdge] c
Syntax::tsntxi: "a\[DirectedEdge]b\[UndirectedEdge]c" is incomplete; more input is needed.
This next one is not uncommon:
FullForm[Hold[a \[LeftTee] b \[UpTee] c]]
FullForm[ToExpression["Hold[a \[LeftTee] b \[UpTee] c]"]]
Hold[LeftTee[a,UpTee[b,c]]]
Hold[UpTee[LeftTee[a,b],c]]
After you try it in a notebook, try it on the command line, too. (There are several differences between the notebook and command line regarding parsing.)
Many operators do not have a FullForm
(unevaluated interpretation) equal to the functions they represent. Consider Divide
:
(* Give the Head of the first element within Hold. *)
Hold[a/b][[1, 0]]
Times
Considering this one fact alone, whatever strategy one uses to inspect an instantiated expression to determine precedence will have to account for every special case of how the FullForm
of each operator might interact with that of another in a nongeneric way. The amount of manual labor one has to do to account for this is $Ω(n^2)$, where $n$ is the number operators.
Then there is the case of GreaterSlantEqual
and LessSlantEqual
, both the symbols and their corresponding functions. The short version is, these two symbols are now disassociated from their corresponding functions, which functions now have no operator notation despite the insistance of the documentation. To be clear, I think it's the right move to remap the *SlantEqual
symbols to >=
and <=
, but...
Conclusion
At the end of the day, there just is no public language specification for Mathematica/Wolfram Language. In a sense, the language is defined to be whatever Mathematica does. And what Mathematica does on the command line may differ from what it does in the notebook, which may differ from the behavior of ToExpression and friends, all of which may differ from the documentation, which itself may be inconsistent—and all of this in just a single version (v11.3) on a single platform (macos).
So the answer is, no, you cannot determine precedence programmatically, or any other way, for that matter.
∣
falls between/
(division) and\[Backslash]
(unused). $\endgroup$Precedence[]
, no? $\endgroup$Precedence @ Divisible
though that's completely inconsistent with what I've determined experimentally. $\endgroup$Precedence
doesn't know aboutDivisible
. It seems to return 670 for any built-in function that doesn't have an operator, and it returns the same forDivisible
. $\endgroup$??∣
also fails. $\endgroup$