Hot answers tagged

38

Symbols are created in the current context during parsing. This should not be a problem in normal circumstances as the symbols are merely "initialized" without values or properties. See these posts for more information: Is it possible to use Begin and End inside a Manipulate? Why doesn't this use of Begin[] work? When does Mathematica create a new Symbol? ...


24

Is it strictly necessary? In most cases, no, but doing so gives us many advantages. Most of those advantages might be worked around with various heuristics. But most importantly, it would have been very difficult to have a robust typesetting system. First, the FrontEnd was always going to need some intermediate representation which is not simply textual. ...


23

First of all, I agree, as OP mentioned in his comment, ANTLR is one of the proper ways to go. Now for this specific task, it might be easier to just compose a parser in the "dirty" way, except we don't have to go so far to regex. In my opinion Mathematica's StringExpression is much more powerful and very suitable for the job. All we have to do is (as OP ...


16

The main points of this answer are that,first, it seems rather difficult to have a fully universal mechanism for option-validation, and second, such a mechanism is not currently available in Mathematica on the language level (meaning automation of complete option-checking, including both the option's name and value). In the particular case in question, ...


15

Not really an answer and I may have to delete it, but...I had something of a different reaction to this business of the front end having its own parser, circa 2004. For motivation I recommend the original. I knew a front end that swallowed a kernel. I don't know why it swallowed a kernel. They taste infernal. I knew a front end that swallowed a ...


14

This answer is based on the Mathematica functional parsers package FunctionalParsers.m used and described in these blog posts. This command loads that package: Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/FunctionalParsers.m"] Using direct parser definitions (first answer) Consider the EBNF: <space> = { '...


14

The Documentation states (emphasis is mine): Pattern (:): s:obj represents the pattern object obj, assigned the name s. Optional (:): p:v is a pattern object that represents an expression of the form p, which, if omitted, should be replaced by v. I think that the Documentation page for Optional is misleading since it states that the infix operator for ...


13

tokenize[str_] := Module[{exp, nb = CreateDocument[{ExpressionCell@ InputForm@MakeExpression[str, StandardForm]}, Visible -> False]}, SelectionMove[nb, Next, Cell]; exp = Flatten[ NotebookRead[nb][[1, 1]] /. {RowBox -> List, i_String /; StringMatchQ[i, Whitespace ..] :> Sequence[]}]; NotebookClose[nb]; exp[[3 ;;-2]]...


13

I am a developer at Wolfram Research and I am trying to share some of the work I have been doing with parsing WL code. I have written a package for parsing WL code and retaining interesting metadata, such as file and line information. I also expose a tokenization function. The paclet is available on the public paclet server: In[1]:= PacletUpdate["AST","...


12

No, AFAIK there is no way to see the FullForm and I think your conclusion is correct. The ^^ is not an operator, it is a form how you can input a number. Effectively, this behavior applies to all form of numerical input. For instance this is unholdable too HoldComplete[16*^2] (* HoldComplete[1600] *) Advanced expanation To give a more thorough ...


12

I worked on Interpreter. As far as the implentation is now, the DelimitedSequence parser does not support quoting, so what you want can't be done. We'll try to add it in a future version.


12

To not mess with $ContextPath you can Block it just for that evaluation. e.g. Block[{$ContextPath = {}}, ...]. But we don't want to do that for the evaluation, input for FullForm should be evaluated freely and only the result should be returned with full symbols' names. To achieve that you should know: $ContextPath and $Context affect the typesetting. ...


12

I'd do the minimum necessary to make a legitimate, unambiguous Mathematica expression, and then let Mathematica rewrite it. stepexpr[s_] := ToExpression[StringReplace[s, {"(" -> " dummy[", ")" -> "]"}]] /. op_Symbol dummy[args__] -> op[args] /. dummy -> List This replaces () expressions with a dummy[] function in the string, making a legal ...


11

Here is how I would approach this problem. First, when possible we should make use of the Attributes of the functions in use with regard to their effect on pattern matching. These attributes follow the natural application of the functions to which they apply so often they make mathematical matching easier. (Sometimes you don't want that.) The OneIdentity ...


11

This can be done using functional parsers. Does the solution below look like something you would be interested in? (If you are I will elaborate on the current answer...) Grammar Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/FunctionalParsers.m"] ebnfToy = " <program> = ( [ <wgap> ] &> <...


11

The notebook interface and the kernel have different parsers, because the notebook needs to manipulate the input before the kernel sees it for a variety of reasons. There are many differences between the two parsers. It has not been a priority to Wolfram to fix the differences. I have been trying to catalog the differences here (rendered as a webpage here)....


10

This seems to work: StringCases["blabla ...Hello Hello ... blabla ... Goobye Goobye ..", Longest[___ ~~ a : "Hello"] ~~ b : Shortest[___ ~~ "Goobye"] :> a ~~ b] Update If there are multiple substrings to extract you can use recursion: extractbetween[str_, x_, y_] := Module[{f}, f[s_] := StringCases[s, Longest[a___ ~~ x] ~~ b : Shortest[___ ~~ ...


10

The result of the front-end's parsing is definitely used, but it does not need to be complete, it seems. This can be seen by constructing Cells/Boxes manually. Try RawBoxes@RowBox[{"1", "+", RowBox[{"2", "*", "3"}]}] RawBoxes@RowBox[{RowBox[{"1", "+", "2"}], "*", "3"}] RawBoxes@RowBox[{"1", "+", "2", "*", "3"}] These all display as 1 + 2 * 3 but you can ...


10

Please also see a related problem here, it might give you more ideas. Words can be banal wrt all words in the dictionary or wrt some particular list of words. Let's consider the later case. WordFrequencyData gives the frequency of word in typical published English text. Within a given list of words you can find the max-frequency word and rescale the rest of ...


9

Recent versions of Mathematica no longer use Java for parsing JSON, and work very fast. There are at least three ways to read JSON. Import[..., "JSON"] is fast an reliable. It returns nested Lists and Rules. Import[..., "RawJSON"] is faster than "JSON" and returns nested Lists and Associations. These are much easier to work with than Rules. The Developer` ...


9

I'm posting this variant in the hopes that it will be a little more educational. Otherwise doesn't add anything over Kuba's version. Generally, parsing can be done using StringCases. You'll need to build up a string expression that describes the pattern of the file name, much the same way you'd write "%d_%s_Polarizer%dDeg-Temp%dK.dpt" when working with ...


9

This is an ideal use case for SemanticImport, but unfortunately it has issues getting the commas right in version 10.0. Luckily, version 10.0.1 has already fixed this bug:


9

I extended the original question to support piping like the following. wget -qO- "http://google.com" // cat and it outputs the following. cat[][wget[-qO-,http://google.com][]] To get the following. CellPrint@ Cell[BoxData[""], "Input", Evaluatable -> True, CellEvaluationFunction -> Function[ Module[{t}, t = List@ ...


9

Note: If you want this type of parser added for JSON, I think WRI could do it easily. They already have Developer`ReadRawJSONStream so I think adding some event-based parsing to that wouldn't be too much to ask This has been kicking around my for the past few days, so here's a quick example of what I discussed in my comment. It is by no means complete, but ...


9

Assuming everything is syntactically correct, the function that does what you want is Internal`ProcessEquations`SeparateEquations: Internal`ProcessEquations`SeparateEquations[{y''[x] == y[x], y[0] == 1, y[1] == 1}, {x}, {y}] (* {{}, {y[0] == 1, y[1] == 1}, {}, {y''[x] == y[x]}} *) Internal`ProcessEquations`SeparateEquations[ {y''[x] - z[x] == 0, y[0] ==...


8

The simple answer is, if you want a string converted to StandardForm, you could wrap BoxData around it. E.g., CellPrint[Cell[BoxData["myFunction::usage=\"myFunction does ...\";"], "Input"]] But, in general, I wouldn't structure this as a question of CellPrint vs. FrontEnd`CellPrint. FrontEnd`CellPrint is undocumented, and therefore there is no contract ...


8

shortestStringCases[str_String, from_String, to_String] := StringCases[ str, (from ~~ mid___ ~~ to) /; StringFreeQ[mid, {from, to}]] shortestStringCases["blah X blah X first Y blah X blah X second Y", "X", "Y"] (* {"X first Y", "X second Y"} *)


8

Just define a new function that could take different inputs, either through overloading or using If, Which, or Switch. Clear[f] f[x_. Power[z, e_.]] := {x, e} f[Times[x_, Power[z, e_.]]] := {x, e} f[x_] := {x, 0} Cases[lis, y : Alternatives[x_., x_. Power[z, e_.], Times[x_, Power[z, e_.]]] :> f[y]] (* {{6, 0}, {1, -4}, {4, -2}, {4, 2}, {1, 4}} *) ...


8

For producing a NB file without line breaks you can Get it as a Notebook expression, then Export it with PageWidth -> Infinity as "Package" (other possible options does not work correctly: Export ignores PageWidth -> Infinity when exporting as "NB" and corrupts the code when exporting as "Text"): Export["document.nb", Get["ExampleData/document.nb"], "...


Only top voted, non community-wiki answers of a minimum length are eligible