6
$\begingroup$

In Mathematica, space can often be used to indicate multiply. For example "a b" = "a * b". On the other hand, in a *.m file, we also have multiple statements separated by newlines. How does the Mathematica parser decide what is "space to mean multiply" vs "space to separate statements" ?

Is there a formal spec of the mathematica grammar somewhere?

Improve this question
$\endgroup$
15
  • 3
    $\begingroup$ After having been burnt a few times by this problem, I try to always end the lines in a way that makes it obvious that the code continues on the next line. For example, in a multiplication add a trailing * at the end of the line, or a + for a continued addition. Also, whenever there are open brackets or parentheses, the parser knows to continue parsing on the next line. I.e., I try to make it obvious instead of relying on parser conventions. $\endgroup$
    – Roman
    Commented Mar 15, 2019 at 2:47
  • 1
    $\begingroup$ @Roman: I agree with your advice. Unfortunately, I'm writing a mathematica parser, so I need to be able to handle whatever insanity the worst mathematica programmer has written up. :-( $\endgroup$
    – none
    Commented Mar 15, 2019 at 3:42
  • 1
    $\begingroup$ From what I can tell, Mathematica treats newlines as implicit multiplication if and only if the statement needs more input (i.e. the cases where not doing so would lead to a "Statement is incomplete, more input is needed" error),e.g. if closing brackets are missing, or the line ends with a binary operator $\endgroup$
    – Lukas Lang
    Commented Mar 15, 2019 at 8:12
  • 1
    $\begingroup$ @Roman That sounds strange. The parser should not move on to the next line if it was able to finish its job at the point of reaching the \n. Perhaps you encountered strange behaviour in a notebook? Notebooks are a bit special because code is represented not as text but as boxes. Sometimes things that look the same have different underlying boxes, leading to surprises. I tried to construct an example where it would appear that there's a newline after a complete expression yet it is still interpreted as multiplication, but I wasn't. I would not be surprised if it is possible though. $\endgroup$
    – Szabolcs
    Commented Mar 15, 2019 at 9:58
  • 1
    $\begingroup$ @Szabolcs you're probably right that it was a notebook, not a package. I'm having a hard time reconstructing an example too. I'm more trying to emit a vague feeling that it isn't as simple as you suggest in your answer. But maybe you're right in the case of *.m files. $\endgroup$
    – Roman
    Commented Mar 15, 2019 at 10:03

2 Answers 2

Reset to default
7
$\begingroup$

Is there a formal spec of the Mathematica grammar somewhere?

No, not any official one. There are unofficial efforts out there, but it is at this point well known in the community that even the different parsers created by Wolfram Research have minor differences between them. For example, xx\[LongDash]xx is parsed differently by the FE and the kernel (in a notebook and on the command line or with ToExpression).

How does the Mathematica parser decide what is "space to mean multiply" vs "space to separate statements" ?

A newline is only interpreted as multiplication if the preceding text cannot be parsed as a complete expression and the parser has a reason to move on to the next line. Example:

There is no multiplication here because a is a complete valid expression:

a
b

There is multiplication in the following because f[a is not complete.

f[
  a
  b
]

There is multiplication here too because the comment was incomplete at the newline, causing the parser to keep going:

a (*
*) b

Finally, newlines can be escaped:

a \
b

This is always tricky because it is not visually clear if the \ is followed by a newline or a space.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You might want to mention the fact that the newline need not be interpreted as implicit multiplication if a line is incomplete, instead it is simply interpreted as a whitespace, which might itself be interpreted as implicit multiplication. (See also my comment under the question) $\endgroup$
    – Lukas Lang
    Commented Mar 16, 2019 at 23:24
2
$\begingroup$

@Szbalocs has answered the syntax question. I'd like to say more about a formal specification for Wolfram Language.

Formal Specification of Wolfram Language

There is no official formal specification from Wolfram. I have a project to create a formal specification for Wolfram Language that I invite anyone and everyone to participate in. You will see that the specification is only in the earliest stages of being written, but there has been significant work by me and others to investigate the behavior of Wolfram Language in its various implementations, much of which is documented here on SE.

Progress So Far

The most complete part of the specification effort is for number representations, but only a few days ago we discovered a couple of edge cases with number forms that have yet to be pushed to the GitHub repo—or published anywhere else.

Significantly, I have cataloged every Wolfram Language operator along with the arity, associativity, precedence, and other information. This information has been unavailable prior to its publication in the repository.

I envision the WLTools GitHub organization as a central repository for resources than can be shared across Wolfram Language Projects. Beyond a language spec, I plan on creating a machine readable repository of test cases. Many of us already maintain our own corpus of tests. We would all benefit from a test suite that covers each of the corner cases all of which no single one of us would have discovered alone.

Challenges of Creating a Formal Specification

I have written about the challenges of creating a formal spec for Wolfram Language in a series of articles—three so far—on my blog. I list only a few:

  1. There are differences not only between different versions of Mathematica, but also between the notebook interface, ToExpression, and the command line interface. For example, my operator catalog covers the behavior of the command line and ToExpression, but I am currently still working on the behavior of the notebook frontend.
  2. The demarcation problem, which manifests in several ways:
  3. Sidestepping the philosophical question of, what is a language?, the only source of data for Wolfram Language are implementations with a great many bugs. Does the formal spec fix the bugs? Does the formal spec fix mistakes in design?
  4. Must Mathematica conform to the formal spec? If the spec allows both - and \[Minus] in scientific notation number forms, code conforming to the spec won't evaluate in Mathematica.

Despite the challenges and inevitable disagreements about where any given boundary should be set, I obviously still think it is a worthwhile endeavor. An imperfect spec is better than the pile of nothing we currently have.

Resources for Wolfram Language Implementors

Since none explicitly mentioned they are writing a parser, here are some resources that will make your work a lot easier:

  • The unofficial spec, obviously.
  • Other Wolfram Language projects and their developer communities.
  • The Wolfram Language Slack Workspace (public invite link) which hosts many brilliant, incredibly knowledgable developers of Wolfram Language tools. We discuss implementation challenges and questions, share resources, and generally have interesting conversations about the technical aspects of Wolfram Language. We have a lot of advice for someone who is just setting out to write a Wolfram Language parser.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.